Prepare for 1.11.1 release.

Since it is not any better by default without initializing the key, we
disable it by default. It can be turned on with JANET_PRF in
janetconf.h.

.builds/meson.yml

@@ -0,0 +1,14 @@
+image: openbsd/latest
+sources:
+- https://git.sr.ht/~bakpakin/janet
+packages:
+- meson
+tasks:
+- build: |
+    cd janet
+    meson setup build --buildtype=release
+    cd build
+    ninja
+    ninja test
+    doas ninja install
+    doas jpm --verbose install circlet

.builds/meson_min.yml

@@ -0,0 +1,23 @@
+image: openbsd/latest
+sources:
+- https://git.sr.ht/~bakpakin/janet
+packages:
+- meson
+tasks:
+- build: |
+    cd janet
+    meson setup build --buildtype=release
+    cd build
+    meson configure -Dsingle_threaded=true
+    meson configure -Dnanbox=false
+    meson configure -Ddynamic_modules=false
+    meson configure -Ddocstrings=false
+    meson configure -Dnet=false
+    meson configure -Dsourcemaps=false
+    meson configure -Dpeg=false
+    meson configure -Dassembler=false
+    meson configure -Dint_types=false
+    meson configure -Dtyped_array=false
+    meson configure -Dreduced_os=true
+    meson configure -Dprf=false
+    ninja # will not pass tests but should build

.gitattributes

@@ -0,0 +1 @@
+*.janet linguist-language=Clojure

CHANGELOG.md

@@ -1,6 +1,109 @@
 # Changelog
 All notable changes to this project will be documented in this file.
 
+## 1.11.1 - 2020-07-25
+- Fix jpm and git with multiple git installs on Windows
+- Fix importing a .so file in the current directory
+- Allow passing byte sequence types directly to typed-array constructors.
+- Fix bug sending files between threads.
+- Disable PRF by default.
+- Update the soname.
+
+## 1.11.0 - 2020-07-18
+- Add `forever` macro.
+- Add `any?` predicate to core.
+- Add `jpm list-pkgs` subcommand to see which package aliases are in the listing.
+- Add `jpm list-installed` subcommand to see which packages are installed.
+- Add `math/int-min`, `math/int-max`, `math/int32-min`, and `math/int32-max` for getting integer limits.
+- The gc interval is now autotuned, to prevent very bad gc behavior.
+- Improvements to the bytecode compiler, Janet will now generate more efficient bytecode.
+- Add `peg/find`, `peg/find-all`, `peg/replace`, and `peg/replace-all`
+- Add `math/nan`
+- Add `forv` macro
+- Add `symbol/slice`
+- Add `keyword/slice`
+- Allow cross compilation with Makefile.
+- Change `compare-primitve` to `cmp` and make it more efficient.
+- Add `reverse!` for reversing an array or buffer in place.
+- `janet_dobytes` and `janet_dostring` return parse errors in \*out
+- Add `repeat` macro for iterating something n times.
+- Add `eachy` (each yield) macro for iterating a fiber.
+- Fix `:generate` verb in loop macro to accept non symbols as bindings.
+- Add `:h`, `:h+`, and `:h*` in `default-peg-grammar` for hexidecimal digits.
+- Fix `%j` formatter to print numbers precisely (using the `%.17g` format string to printf).
+
+## 1.10.1 - 2020-06-18
+- Expose `janet_table_clear` in API.
+- Respect `JANET_NO_PROCESSES` define when building
+- Fix `jpm` rules having multiple copies of the same dependency.
+- Fix `jpm` install in some cases.
+- Add `array/trim` and `buffer/trim` to shrink the backing capacity of these types
+  to their current length.
+
+## 1.10.0 - 2020-06-14
+- Hardcode default jpm paths on install so env variables are needed in fewer cases.
+- Add `:no-compile` to `create-executable` option for jpm.
+- Fix bug with the `trace` function.
+- Add `:h`, `:a`, and `:c` flags to `thread/new` for creating new kinds of threads.
+  By default, threads will now consume much less memory per thread, but sending data between
+  threads may cost more.
+- Fix flychecking when using the `use` macro.
+- CTRL-C no longer exits the repl, and instead cancels the current form.
+- Various small bug fixes
+- New MSI installer instead of NSIS based installer.
+- Make `os/realpath` work on windows.
+- Add polymorphic `compare` functions for comparing numbers.
+- Add `to` and `thru` peg combinators.
+- Add `JANET_GIT` environment variable to jpm to use a specific git binary (useful mainly on windows).
+- `asm` and `disasm` functions now use keywords instead of macros for keys. Also
+  some slight changes to the way constants are encoded (remove wrapping `quote` in some cases).
+- Expose current macro form inside macros as (dyn :macro-form)
+- Add `tracev` macro.
+- Fix compiler bug that emitted incorrect code in some cases for while loops that create closures.
+- Add `:fresh` option to `(import ...)` to overwrite the module cache.
+- `(range x y 0)` will return an empty array instead of hanging forever.
+- Rename `jpm repl` to `jpm debug-repl`.
+
+## 1.9.1 - 2020-05-12
+- Add :prefix option to declare-source
+- Re-enable minimal builds with the debugger.
+- Add several flags for configuring Janet on different platforms.
+- Fix broken meson build from 1.9.0 and add meson to CI.
+- Fix compilation issue when nanboxing is disabled.
+
+## 1.9.0 - 2020-05-10
+- Add `:ldflags` option to many jpm declare functions.
+- Add `errorf` to core.
+- Add `lenprefix` combinator to PEGs.
+- Add `%M`, `%m`, `%N`, and `%n` formatters to formatting functions. These are the
+  same as `%Q`, `%q`, `%P`, and `%p`, but will not truncate long values.
+- Add `fiber/root`.
+- Add beta `net/` module to core for socket based networking.
+- Add the `parse` function to parse strings of source code more conveniently.
+- Add `jpm rule-tree` subcommand.
+- Add `--offline` flag to jpm to force use of the cache.
+- Allow sending pointers and C functions across threads via `thread/send`.
+- Fix bug in `getline`.
+- Add `sh-rule` and `sh-phony` to jpm's dialect of Janet.
+- Change C api's `janet_formatb` -> `janet_formatbv`, and add new function `janet_formatb` to C api.
+- Add `edefer` macro to core.
+- A struct/table literal/constructor with duplicate keys will use the last value given.
+  Previously, this was inconsistent between tables and structs, literals and constructor functions.
+- Add debugger to core. The debugger functions are only available
+  in a debug repl, and are prefixed by a `.`.
+- Add `sort-by` and `sorted-by` to core.
+- Support UTF-8 escapes in strings via `\uXXXX` or `\UXXXXXX`.
+- Add `math/erf`
+- Add `math/erfc`
+- Add `math/log1p`
+- Add `math/next`
+- Add os/umask
+- Add os/perm-int
+- Add os/perm-string
+- Add :int-permissions option for os/stat.
+- Add `jpm repl` subcommand, as well as `post-deps` macro in project.janet files.
+- Various bug fixes.
+
 ## 1.8.1 - 2020-03-31
 - Fix bugs for big endian systems
 - Fix 1.8.0 regression on BSDs

CONTRIBUTING.md

@@ -35,8 +35,9 @@ may require changes before being merged.
   [astyle](http://astyle.sourceforge.net/astyle.html). You will probably need
   to install this, but it can be installed with most package managers.
 
-  For janet code, the use lisp indentation with 2 spaces. One can use janet.vim to
-  do this indentation, or approximate as close as possible.
+  For janet code, use lisp indentation with 2 spaces. One can use janet.vim to
+  do this indentation, or approximate as close as possible. There is a janet formatter
+  in [spork](https://github.com/janet-lang/spork.git) that can be used to format code as well.
 
 ## C style
 

Makefile

@@ -33,12 +33,20 @@ JANET_TARGET=build/janet
 JANET_LIBRARY=build/libjanet.so
 JANET_STATIC_LIBRARY=build/libjanet.a
 JANET_PATH?=$(LIBDIR)/janet
-MANPATH?=$(PREFIX)/share/man/man1/
-PKG_CONFIG_PATH?=$(LIBDIR)/pkgconfig
+JANET_MANPATH?=$(PREFIX)/share/man/man1/
+JANET_PKG_CONFIG_PATH?=$(LIBDIR)/pkgconfig
 DEBUGGER=gdb
+SONAME_SETTER=-Wl,-soname,
 
-CFLAGS:=$(CFLAGS) -std=c99 -Wall -Wextra -Isrc/include -Isrc/conf -fPIC -O2 -fvisibility=hidden
-LDFLAGS:=$(LDFLAGS) -rdynamic
+# For cross compilation
+HOSTCC?=$(CC)
+HOSTAR?=$(AR)
+CFLAGS?=-fPIC -O2
+LDFLAGS?=-rdynamic
+
+COMMON_CFLAGS:=-std=c99 -Wall -Wextra -Isrc/include -Isrc/conf -fvisibility=hidden
+BOOT_CFLAGS:=-DJANET_BOOTSTRAP -DJANET_BUILD=$(JANET_BUILD) -O0 -g $(COMMON_CFLAGS)
+BUILD_CFLAGS:=$(CFLAGS) $(COMMON_CFLAGS)
 
 # For installation
 LDCONFIG:=ldconfig "$(LIBDIR)"
@@ -47,6 +55,7 @@ LDCONFIG:=ldconfig "$(LIBDIR)"
 UNAME:=$(shell uname -s)
 ifeq ($(UNAME), Darwin)
 	CLIBS:=$(CLIBS) -ldl
+	SONAME_SETTER:=-Wl,-install_name,
 	LDCONFIG:=true
 else ifeq ($(UNAME), Linux)
 	CLIBS:=$(CLIBS) -lrt -ldl
@@ -94,6 +103,7 @@ JANET_CORE_SOURCES=src/core/abstract.c \
 				   src/core/io.c \
 				   src/core/marsh.c \
 				   src/core/math.c \
+				   src/core/net.c \
 				   src/core/os.c \
 				   src/core/parse.c \
 				   src/core/peg.c \
@@ -128,7 +138,6 @@ JANET_BOOT_HEADERS=src/boot/tests.h
 ##########################################################
 
 JANET_BOOT_OBJECTS=$(patsubst src/%.c,build/%.boot.o,$(JANET_CORE_SOURCES) $(JANET_BOOT_SOURCES))
-BOOT_CFLAGS:=-DJANET_BOOTSTRAP -DJANET_BUILD=$(JANET_BUILD) $(CFLAGS)
 
 $(JANET_BOOT_OBJECTS): $(JANET_BOOT_HEADERS)
 
@@ -146,6 +155,8 @@ build/janet.c: build/janet_boot src/boot/boot.janet
 ##### Amalgamation #####
 ########################
 
+SONAME=libjanet.so.1.11
+
 build/shell.c: src/mainclient/shell.c
 	cp $< $@
 
@@ -156,24 +167,26 @@ build/janetconf.h: src/conf/janetconf.h
 	cp $< $@
 
 build/janet.o: build/janet.c build/janet.h build/janetconf.h
-	$(CC) $(CFLAGS) -c $< -o $@ -I build
+	$(HOSTCC) $(BUILD_CFLAGS) -c $< -o $@ -I build
 
 build/shell.o: build/shell.c build/janet.h build/janetconf.h
-	$(CC) $(CFLAGS) -c $< -o $@ -I build
+	$(HOSTCC) $(BUILD_CFLAGS) -c $< -o $@ -I build
 
 $(JANET_TARGET): build/janet.o build/shell.o
-	$(CC) $(LDFLAGS) $(CFLAGS) -o $@ $^ $(CLIBS)
+	$(HOSTCC) $(LDFLAGS) $(BUILD_CFLAGS) -o $@ $^ $(CLIBS)
 
 $(JANET_LIBRARY): build/janet.o build/shell.o
-	$(CC) $(LDFLAGS) $(CFLAGS) -shared -o $@ $^ $(CLIBS)
+	$(HOSTCC) $(LDFLAGS) $(BUILD_CFLAGS) $(SONAME_SETTER)$(SONAME) -shared -o $@ $^ $(CLIBS)
 
 $(JANET_STATIC_LIBRARY): build/janet.o build/shell.o
-	$(AR) rcs $@ $^
+	$(HOSTAR) rcs $@ $^
 
 ###################
 ##### Testing #####
 ###################
 
+# Testing assumes HOSTCC=CC
+
 TEST_SCRIPTS=$(wildcard test/suite*.janet)
 
 repl: $(JANET_TARGET)
@@ -190,12 +203,12 @@ valgrind: $(JANET_TARGET)
 test: $(JANET_TARGET) $(TEST_PROGRAMS)
 	for f in test/suite*.janet; do ./$(JANET_TARGET) "$$f" || exit; done
 	for f in examples/*.janet; do ./$(JANET_TARGET) -k "$$f"; done
-	./$(JANET_TARGET) -k auxbin/jpm
+	./$(JANET_TARGET) -k jpm
 
 valtest: $(JANET_TARGET) $(TEST_PROGRAMS)
 	for f in test/suite*.janet; do $(VALGRIND_COMMAND) ./$(JANET_TARGET) "$$f" || exit; done
 	for f in examples/*.janet; do ./$(JANET_TARGET) -k "$$f"; done
-	$(VALGRIND_COMMAND) ./$(JANET_TARGET) -k auxbin/jpm
+	$(VALGRIND_COMMAND) ./$(JANET_TARGET) -k jpm
 
 callgrind: $(JANET_TARGET)
 	for f in test/suite*.janet; do valgrind --tool=callgrind ./$(JANET_TARGET) "$$f" || exit; done
@@ -209,7 +222,7 @@ dist: build/janet-dist.tar.gz
 build/janet-%.tar.gz: $(JANET_TARGET) \
 	src/include/janet.h src/conf/janetconf.h \
 	jpm.1 janet.1 LICENSE CONTRIBUTING.md $(JANET_LIBRARY) $(JANET_STATIC_LIBRARY) \
-	build/doc.html README.md build/janet.c build/shell.c auxbin/jpm
+	build/doc.html README.md build/janet.c build/shell.c jpm
 	$(eval JANET_DIST_DIR = "janet-$(shell basename $*)")
 	mkdir -p build/$(JANET_DIST_DIR)
 	cp -r $^ build/$(JANET_DIST_DIR)/
@@ -228,7 +241,9 @@ build/doc.html: $(JANET_TARGET) tools/gendoc.janet
 ##### Installation #####
 ########################
 
-SONAME=libjanet.so.1
+build/jpm: jpm $(JANET_TARGET)
+	$(JANET_TARGET) tools/patch-jpm.janet jpm build/jpm "--libpath=$(LIBDIR)" "--headerpath=$(INCLUDEDIR)/janet" "--binpath=$(BINDIR)"
+	chmod +x build/jpm
 
 .INTERMEDIATE: build/janet.pc
 build/janet.pc: $(JANET_TARGET)
@@ -242,10 +257,10 @@ build/janet.pc: $(JANET_TARGET)
 	echo "Description: Library for the Janet programming language." >> $@
 	$(JANET_TARGET) -e '(print "Version: " janet/version)' >> $@
 	echo 'Cflags: -I$${includedir}' >> $@
-	echo 'Libs: -L$${libdir} -ljanet $(LDFLAGS)' >> $@
+	echo 'Libs: -L$${libdir} -ljanet' >> $@
 	echo 'Libs.private: $(CLIBS)' >> $@
 
-install: $(JANET_TARGET) build/janet.pc
+install: $(JANET_TARGET) build/janet.pc build/jpm
 	mkdir -p '$(DESTDIR)$(BINDIR)'
 	cp $(JANET_TARGET) '$(DESTDIR)$(BINDIR)/janet'
 	mkdir -p '$(DESTDIR)$(INCLUDEDIR)/janet'
@@ -256,12 +271,12 @@ install: $(JANET_TARGET) build/janet.pc
 	cp $(JANET_STATIC_LIBRARY) '$(DESTDIR)$(LIBDIR)/libjanet.a'
 	ln -sf $(SONAME) '$(DESTDIR)$(LIBDIR)/libjanet.so'
 	ln -sf libjanet.so.$(shell $(JANET_TARGET) -e '(print janet/version)') $(DESTDIR)$(LIBDIR)/$(SONAME)
-	cp -rf auxbin/* '$(DESTDIR)$(BINDIR)'
-	mkdir -p '$(DESTDIR)$(MANPATH)'
-	cp janet.1 '$(DESTDIR)$(MANPATH)'
-	cp jpm.1 '$(DESTDIR)$(MANPATH)'
-	mkdir -p '$(DESTDIR)$(PKG_CONFIG_PATH)'
-	cp build/janet.pc '$(DESTDIR)$(PKG_CONFIG_PATH)/janet.pc'
+	cp -rf build/jpm '$(DESTDIR)$(BINDIR)'
+	mkdir -p '$(DESTDIR)$(JANET_MANPATH)'
+	cp janet.1 '$(DESTDIR)$(JANET_MANPATH)'
+	cp jpm.1 '$(DESTDIR)$(JANET_MANPATH)'
+	mkdir -p '$(DESTDIR)$(JANET_PKG_CONFIG_PATH)'
+	cp build/janet.pc '$(DESTDIR)$(JANET_PKG_CONFIG_PATH)/janet.pc'
 	[ -z '$(DESTDIR)' ] && $(LDCONFIG) || true
 
 uninstall:
@@ -269,9 +284,9 @@ uninstall:
 	-rm '$(DESTDIR)$(BINDIR)/jpm'
 	-rm -rf '$(DESTDIR)$(INCLUDEDIR)/janet'
 	-rm -rf '$(DESTDIR)$(LIBDIR)'/libjanet.*
-	-rm '$(DESTDIR)$(PKG_CONFIG_PATH)/janet.pc'
-	-rm '$(DESTDIR)$(MANPATH)/janet.1'
-	-rm '$(DESTDIR)$(MANPATH)/jpm.1'
+	-rm '$(DESTDIR)$(JANET_PKG_CONFIG_PATH)/janet.pc'
+	-rm '$(DESTDIR)$(JANET_MANPATH)/janet.1'
+	-rm '$(DESTDIR)$(JANET_MANPATH)/jpm.1'
 	# -rm -rf '$(DESTDIR)$(JANET_PATH)'/* - err on the side of correctness here
 
 #################
@@ -287,6 +302,7 @@ build/janet.tmLanguage: tools/tm_lang_gen.janet $(JANET_TARGET)
 
 clean:
 	-rm -rf build vgcore.* callgrind.*
+	-rm -rf test/install/build test/install/modpath
 
 test-install:
 	cd test/install \
@@ -296,10 +312,33 @@ test-install:
 		&& build/testexec \
 		&& jpm --verbose quickbin testexec.janet build/testexec2 \
 		&& build/testexec2 \
-		&& jpm --verbose --testdeps --modpath=. install https://github.com/janet-lang/json.git
-	cd test/install && jpm --verbose --test --modpath=. install https://github.com/janet-lang/jhydro.git
-	cd test/install && jpm --verbose --test --modpath=. install https://github.com/janet-lang/path.git
-	cd test/install && jpm --verbose --test --modpath=. install https://github.com/janet-lang/argparse.git
+		&& mkdir -p modpath \
+		&& jpm --verbose --testdeps --modpath=./modpath install https://github.com/janet-lang/json.git
+	cd test/install && jpm --verbose --test --modpath=./modpath install https://github.com/janet-lang/jhydro.git
+	cd test/install && jpm --verbose --test --modpath=./modpath install https://github.com/janet-lang/path.git
+	cd test/install && jpm --verbose --test --modpath=./modpath install https://github.com/janet-lang/argparse.git
+
+help:
+	@echo
+	@echo 'Janet: A Dynamic Language & Bytecode VM'
+	@echo
+	@echo Usage:
+	@echo '   make            Build Janet'
+	@echo '   make repl       Start a REPL from a built Janet'
+	@echo
+	@echo '   make test       Test a built Janet'
+	@echo '   make valgrind   Assess Janet with Valgrind'
+	@echo '   make callgrind  Assess Janet with Valgrind, using Callgrind'
+	@echo '   make valtest    Run the test suite with Valgrind to check for memory leaks'
+	@echo '   make dist       Create a distribution tarball'
+	@echo '   make docs       Generate documentation'
+	@echo '   make debug      Run janet with GDB or LLDB'
+	@echo '   make install    Install into the current filesystem'
+	@echo '   make uninstall  Uninstall from the current filesystem'
+	@echo '   make clean      Clean intermediate build artifacts'
+	@echo "   make format     Format Janet's own source files"
+	@echo '   make grammar    Generate a TextMate language grammar'
+	@echo
 
 .PHONY: clean install repl debug valgrind test \
-	valtest emscripten dist uninstall docs grammar format
+	valtest dist uninstall docs grammar format help

README.md

@@ -2,13 +2,15 @@
  
 [![Appveyor Status](https://ci.appveyor.com/api/projects/status/bjraxrxexmt3sxyv/branch/master?svg=true)](https://ci.appveyor.com/project/bakpakin/janet/branch/master)
 [![Build Status](https://travis-ci.org/janet-lang/janet.svg?branch=master)](https://travis-ci.org/janet-lang/janet)
-[![builds.sr.ht status](https://builds.sr.ht/~bakpakin/janet/freebsd.yml.svg)](https://builds.sr.ht/~bakpakin/janet/freebsd.yml?)
-[![builds.sr.ht status](https://builds.sr.ht/~bakpakin/janet/openbsd.yml.svg)](https://builds.sr.ht/~bakpakin/janet/openbsd.yml?)
+[![builds.sr.ht status](https://builds.sr.ht/~bakpakin/janet/freebsd.yml.svg)](https://builds.sr.ht/~bakpakin/janet/commits/freebsd.yml?)
+[![builds.sr.ht status](https://builds.sr.ht/~bakpakin/janet/openbsd.yml.svg)](https://builds.sr.ht/~bakpakin/janet/commits/openbsd.yml?)
+[![builds.sr.ht status](https://builds.sr.ht/~bakpakin/janet/meson.yml.svg)](https://builds.sr.ht/~bakpakin/janet/commits/meson.yml?)
+[![builds.sr.ht status](https://builds.sr.ht/~bakpakin/janet/meson_min.yml.svg)](https://builds.sr.ht/~bakpakin/janet/commits/meson_min.yml?)
 
 <img src="https://raw.githubusercontent.com/janet-lang/janet/master/assets/janet-w200.png" alt="Janet logo" width=200 align="left">
 
 **Janet** is a functional and imperative programming language and bytecode interpreter. It is a
-modern lisp, but lists are replaced
+lisp-like language, but lists are replaced
 by other data structures (arrays, tables (hash table), struct (immutable hash table), tuples).
 The language also supports bridging to native code written in C, meta-programming with macros, and bytecode assembly.
 
@@ -35,7 +37,7 @@ Lua, but smaller than GNU Guile or Python.
 * Mutable and immutable arrays (array/tuple)
 * Mutable and immutable hashtables (table/struct)
 * Mutable and immutable strings (buffer/string)
-* Lisp Macros
+* Macros
 * Byte code interpreter with an assembly interface, as well as bytecode verification
 * Tailcall Optimization
 * Direct interop with C via abstract types and C functions
@@ -75,7 +77,7 @@ the SourceHut mirror is actively maintained.
 
 ## Building
 
-### macos and Unix-like
+### macOS and Unix-like
 
 The Makefile is non-portable and requires GNU-flavored make.
 
@@ -86,6 +88,8 @@ make test
 make repl
 ```
 
+Find out more about the available make targets by running `make help`.
+
 ### 32-bit Haiku
 
 32-bit Haiku build instructions are the same as the unix-like build instructions,
@@ -118,6 +122,13 @@ gmake repl
 3. Run `build_win` to compile janet.
 4. Run `build_win test` to make sure everything is working.
 
+To build an `.msi` installer executable, in addition to the above steps, you will have to:
+
+5. Install, or otherwise add to your PATH the [WiX 3.11 Toolset](https://github.com/wixtoolset/wix3/releases)
+6. run `build_win dist`
+
+Now you should have an `.msi`. You can run `build_win install` to install the `.msi`, or execute the file itself.
+
 ### Meson
 
 Janet also has a build file for [Meson](https://mesonbuild.com/), a cross platform build
@@ -135,6 +146,7 @@ cd janet
 meson setup build \
           --buildtype release \
           --optimization 2 \
+          --libdir /usr/local/lib \
           -Dgit_hash=$(git log --pretty=format:'%h' -n 1)
 ninja -C build
 
@@ -200,7 +212,7 @@ If installed, you can also run `man janet` and `man jpm` to get usage informatio
 Janet can be embedded in a host program very easily. The normal build
 will create a file `build/janet.c`, which is a single C file
 that contains all the source to Janet. This file, along with
-`src/include/janet.h` and `src/include/janetconf.h` can dragged into any C
+`src/include/janet.h` and `src/conf/janetconf.h` can be dragged into any C
 project and compiled into the project. Janet should be compiled with `-std=c99`
 on most compilers, and will need to be linked to the math library, `-lm`, and
 the dynamic linker, `-ldl`, if one wants to be able to load dynamic modules. If
@@ -220,10 +232,10 @@ Alternatively, check out [the #janet channel on Freenode](https://webchat.freeno
 
 ## FAQ
 
-### Why is my terminal is spitting out junk when I run the repl?
+### Why is my terminal spitting out junk when I run the repl?
 
 Make sure your terminal supports ANSI escape codes. Most modern terminals will
-support these, but some older terminals, windows consoles, or embedded terminals
+support these, but some older terminals, Windows consoles, or embedded terminals
 will not. If your terminal does not support ANSI escape codes, run the repl with
 the `-n` flag, which disables color output. You can also try the `-s` if further issues
 ensue.

appveyor.yml

@@ -4,7 +4,6 @@ image:
 - Visual Studio 2019
 configuration:
 - Release
-- Debug
 platform:
 - x64
 - x86
@@ -20,10 +19,6 @@ init:
 
 install:
     - set JANET_BUILD=%appveyor_repo_commit:~0,7%
-    - choco install nsis -y -pre --version 3.05
-    # Replace makensis.exe and files with special long string build. This should
-    # prevent issues when setting PATH during installation.
-    - 7z e "tools\nsis-3.05-strlen_8192.zip" -o"C:\Program Files (x86)\NSIS\" -y
     - build_win all
     - refreshenv
     # We need to reload vcvars after refreshing
@@ -49,7 +44,7 @@ artifacts:
     - name: "janet-$(janet_outname)-windows-%platform%"
       path: dist
       type: Zip
-    - path: "janet-$(janet_outname)-windows-%platform%-installer.exe"
+    - path: "janet-$(janet_outname)-windows-%platform%-installer.msi"
       type: File
 
 deploy:

build_win.bat

@@ -116,23 +116,34 @@ copy src\include\janet.h dist\janet.h
 copy src\conf\janetconf.h dist\janetconf.h
 copy build\libjanet.lib dist\libjanet.lib
 
-copy auxbin\jpm dist\jpm
+copy .\jpm dist\jpm
 copy tools\jpm.bat dist\jpm.bat
 
 @rem Create installer
 janet.exe -e "(->> janet/version (peg/match ''(* :d+ `.` :d+ `.` :d+)) first print)" > build\version.txt
-janet.exe -e "(print (= (os/arch) :x64))" > build\64bit.txt
+janet.exe -e "(print (os/arch))" > build\arch.txt
 set /p JANET_VERSION= < build\version.txt
-set /p SIXTYFOUR= < build\64bit.txt
+set /p BUILDARCH= < build\arch.txt
 echo "JANET_VERSION is %JANET_VERSION%"
-"C:\Program Files (x86)\NSIS\makensis.exe" /DVERSION=%JANET_VERSION% /DSIXTYFOUR=%SIXTYFOUR% janet-installer.nsi 
+if defined APPVEYOR_REPO_TAG_NAME (
+    set RELEASE_VERSION=%APPVEYOR_REPO_TAG_NAME%
+) else (
+    set RELEASE_VERSION=%JANET_VERSION%
+)
+if defined CI (
+    set WIXBIN="c:\Program Files (x86)\WiX Toolset v3.11\bin\"
+) else (
+    set WIXBIN=
+)
+%WIXBIN%candle.exe tools\msi\janet.wxs -arch %BUILDARCH% -out build\
+%WIXBIN%light.exe "-sice:ICE38" -b tools\msi -ext WixUIExtension build\janet.wixobj -out janet-%RELEASE_VERSION%-windows-%BUILDARCH%-installer.msi
 exit /b 0
 
 @rem Run the installer. (Installs to the local user with default settings)
 :INSTALL
-@echo Running Installer...
-FOR %%a in (janet-*-windows-*-installer.exe) DO (
-    %%a /S /CurrentUser
+FOR %%a in (janet-*-windows-*-installer.msi) DO (
+    @echo Running Installer %%a...
+    %%a /QN
 )
 exit /b 0
 
@@ -157,6 +168,8 @@ call jpm --verbose --test --modpath=. install https://github.com/janet-lang/path
 @if errorlevel 1 goto :TESTINSTALLFAIL
 call jpm --verbose --test --modpath=. install https://github.com/janet-lang/argparse.git
 @if errorlevel 1 goto :TESTINSTALLFAIL
+call jpm --verbose --modpath=. install https://github.com/bakpakin/x43bot.git
+@if errorlevel 1 goto :TESTINSTALLFAIL
 popd
 exit /b 0
 

examples/debugger.janet

@@ -1,20 +1,18 @@
 ###
 ### A useful debugger library for Janet. Should be used
-### inside a debug repl.
+### inside a debug repl. This has been moved into the core.
 ###
 
 (defn .fiber
   "Get the current fiber being debugged."
   []
-  (if-let [entry (dyn '_fiber)]
-    (entry :value)
-    (dyn :fiber)))
+  (dyn :fiber))
 
 (defn .stack
   "Print the current fiber stack"
   []
   (print)
-  (debug/stacktrace (.fiber) "")
+  (with-dyns [:err-color false] (debug/stacktrace (.fiber) ""))
   (print))
 
 (defn .frame

examples/numarray/numarray.c

@@ -23,7 +23,7 @@ static int num_array_gc(void *p, size_t s) {
     return 0;
 }
 
-Janet num_array_get(void *p, Janet key);
+int num_array_get(void *p, Janet key, Janet *out);
 void num_array_put(void *p, Janet key, Janet value);
 
 static const JanetAbstractType num_array_type = {
@@ -31,7 +31,8 @@ static const JanetAbstractType num_array_type = {
     num_array_gc,
     NULL,
     num_array_get,
-    num_array_put
+    num_array_put,
+    JANET_ATEND_PUT
 };
 
 static Janet num_array_new(int32_t argc, Janet *argv) {
@@ -81,21 +82,20 @@ static const JanetMethod methods[] = {
     {NULL, NULL}
 };
 
-Janet num_array_get(void *p, Janet key) {
+int num_array_get(void *p, Janet key, Janet *out) {
     size_t index;
-    Janet value;
     num_array *array = (num_array *)p;
     if (janet_checktype(key, JANET_KEYWORD))
-        return janet_getmethod(janet_unwrap_keyword(key), methods);
+        return janet_getmethod(janet_unwrap_keyword(key), methods, out);
     if (!janet_checkint(key))
         janet_panic("expected integer key");
     index = (size_t)janet_unwrap_integer(key);
     if (index >= array->size) {
-        value = janet_wrap_nil();
+        return 0;
     } else {
-        value = janet_wrap_number(array->data[index]);
+        *out = janet_wrap_number(array->data[index]);
     }
-    return value;
+    return 1;
 }
 
 static const JanetReg cfuns[] = {

examples/numarray/project.janet

@@ -0,0 +1,7 @@
+(declare-project
+  :name "numarray"
+  :description "Example c lib with abstract type")
+
+(declare-native
+  :name "numarray"
+  :source @["numarray.c"])

examples/numarray/test/numarray_tests.janet

@@ -1,10 +1,4 @@
-(import cook)
-
-(cook/make-native
-    :name "numarray"
-    :source @["numarray.c"])
-
-(import build/numarray :as numarray)
+(import build/numarray)
 
 (def a (numarray/new 30))
 (print (get a 20))

examples/tcpclient.janet

@@ -0,0 +1,6 @@
+(with [conn (net/connect "127.0.0.1" "8000")]
+  (printf "Connected to %q!" conn)
+  (:write conn "Echo...")
+  (print "Wrote to connection...")
+  (def res (:read conn 1024))
+  (pp res))

examples/tcpserver.janet

@@ -0,0 +1,13 @@
+(defn handler
+  "Simple handler for connections."
+  [stream]
+  (defer (:close stream)
+    (def id (gensym))
+    (def b @"")
+    (print "Connection " id "!")
+    (while (:read stream 1024 b)
+      (:write stream b)
+      (buffer/clear b))
+    (printf "Done %v!" id)))
+
+(net/server "127.0.0.1" "8000" handler)

janet-installer.nsi

@@ -1,216 +0,0 @@
-# This file is invoked by build_win.bat
-# Relevant configuration variables are set there.
-
-Unicode True
-
-!echo "Program Files: ${PROGRAMFILES}"
-!addplugindir "tools\"
-
-# Version
-!define PRODUCT_VERSION "${VERSION}.0"
-VIProductVersion "${PRODUCT_VERSION}"
-VIFileVersion "${PRODUCT_VERSION}"
-
-# Use the modern UI
-!define MULTIUSER_EXECUTIONLEVEL Highest
-!define MULTIUSER_MUI
-!define MULTIUSER_INSTALLMODE_COMMANDLINE
-!define MULTIUSER_INSTALLMODE_DEFAULT_REGISTRY_KEY "Software\Janet\${VERSION}"
-!define MULTIUSER_INSTALLMODE_DEFAULT_REGISTRY_VALUENAME ""
-!define MULTIUSER_INSTALLMODE_INSTDIR_REGISTRY_KEY "Software\Janet\${VERSION}"
-!define MULTIUSER_INSTALLMODE_INSTDIR_REGISTRY_VALUENAME ""
-!define MULTIUSER_INSTALLMODE_INSTDIR "Janet-${VERSION}"
-
-!if ${SIXTYFOUR} == "true"
-    !define MULTIUSER_USE_PROGRAMFILES64
-    !define PLATNAME "x64"
-!else
-    !define PLATNAME "x86"
-!endif
-
-# Includes
-!include "MultiUser.nsh"
-!include "MUI2.nsh"
-!include "LogicLib.nsh"
-
-# Basics
-Name "Janet"
-
-# Do some NSIS-fu to figure out at compile time if we are in appveyor
-!define OUTNAME $%APPVEYOR_REPO_TAG_NAME%
-!define "CHECK_${OUTNAME}"
-!define DOLLAR "$"
-!ifdef CHECK_${DOLLAR}%APPVEYOR_REPO_TAG_NAME%
-    # We are not in the appveyor environment, use version name
-    !define OUTNAME_PART ${VERSION}
-!else
-    # We are in appveyor, use git tag name for installer
-    !define OUTNAME_PART ${OUTNAME}
-!endif
-OutFile "janet-${OUTNAME_PART}-windows-${PLATNAME}-installer.exe"
-
-# Some Configuration
-!define APPNAME "Janet"
-!define DESCRIPTION "The Janet Programming Language"
-!define HELPURL "http://janet-lang.org"
-BrandingText "The Janet Programming Language"
-
-# Macros for setting registry values
-!define UNINST_KEY "Software\Microsoft\Windows\CurrentVersion\Uninstall\Janet-${VERSION}"
-!macro WriteEnv key value
-    ${If} $MultiUser.InstallMode == "AllUsers"
-        WriteRegExpandStr HKLM "SYSTEM\CurrentControlSet\Control\Session Manager\Environment" "${key}" "${value}"
-    ${Else}
-        WriteRegExpandStr HKCU "Environment" "${key}" "${value}"
-    ${EndIf}
-!macroend
-!macro DelEnv key
-    ${If} $MultiUser.InstallMode == "AllUsers"
-        DeleteRegValue HKLM "SYSTEM\CurrentControlSet\Control\Session Manager\Environment" "${key}"
-    ${Else}
-        DeleteRegValue HKCU "Environment" "${key}"
-    ${EndIf}
-!macroend
-
-# MUI Configuration
-!define MUI_ICON "assets\icon.ico"
-!define MUI_UNICON "assets\icon.ico"
-!define MUI_HEADERIMAGE
-!define MUI_HEADERIMAGE_BITMAP "assets\janet-w200.png"
-!define MUI_HEADERIMAGE_RIGHT
-!define MUI_ABORTWARNING
-
-# Show a welcome page first
-!insertmacro MUI_PAGE_WELCOME
-!insertmacro MUI_PAGE_LICENSE "LICENSE"
-
-# Pick Install Directory
-!insertmacro MULTIUSER_PAGE_INSTALLMODE
-!insertmacro MUI_PAGE_DIRECTORY
-!insertmacro MUI_PAGE_INSTFILES
-
-# Done
-!insertmacro MUI_PAGE_FINISH
-
-# Need to set a language.
-!insertmacro MUI_LANGUAGE "English"
-
-function .onInit
-    !insertmacro MULTIUSER_INIT
-functionEnd
-
-section "Janet" BfWSection
-
-    createDirectory "$INSTDIR\Library"
-    createDirectory "$INSTDIR\C"
-    createDirectory "$INSTDIR\bin"
-    createDirectory "$INSTDIR\docs"
-    setOutPath "$INSTDIR"
-
-    # Bin files
-    file /oname=bin\janet.exe dist\janet.exe
-    file /oname=logo.ico assets\icon.ico
-    file /oname=bin\jpm.janet auxbin\jpm
-    file /oname=bin\jpm.bat tools\jpm.bat
-
-    # C headers and library files
-    file /oname=C\janet.h dist\janet.h
-    file /oname=C\janetconf.h dist\janetconf.h
-    file /oname=C\janet.lib dist\janet.lib
-    file /oname=C\janet.exp dist\janet.exp
-    file /oname=C\janet.c dist\janet.c
-    file /oname=C\libjanet.lib dist\libjanet.lib
-
-    # Documentation
-    file /oname=docs\docs.html dist\doc.html
-
-    # Other
-    file README.md
-    file LICENSE
-
-    # Uninstaller - See function un.onInit and section "uninstall" for configuration
-    writeUninstaller "$INSTDIR\uninstall.exe"
-
-    # Start Menu
-    createShortCut "$SMPROGRAMS\Janet.lnk" "$INSTDIR\bin\janet.exe" "" "$INSTDIR\logo.ico"
-
-    # Update path
-    ${If} $MultiUser.InstallMode == "AllUsers"
-        EnVar::SetHKLM
-    ${Else}
-        EnVar::SetHKCU
-    ${EndIf}
-    EnVar::AddValue "PATH" "$INSTDIR\bin"
-    Pop $0
-
-    # Set up Environment variables
-    !insertmacro WriteEnv JANET_PATH "$INSTDIR\Library"
-    !insertmacro WriteEnv JANET_HEADERPATH "$INSTDIR\C"
-    !insertmacro WriteEnv JANET_LIBPATH "$INSTDIR\C"
-    !insertmacro WriteEnv JANET_BINPATH "$INSTDIR\bin"
-
-    SendMessage ${HWND_BROADCAST} ${WM_WININICHANGE} 0 "STR:Environment" /TIMEOUT=5000
-
-    # Registry information for add/remove programs
-    WriteRegStr SHCTX "${UNINST_KEY}" "DisplayName" "Janet"
-    WriteRegStr SHCTX "${UNINST_KEY}" "InstallLocation" "$INSTDIR"
-    WriteRegStr SHCTX "${UNINST_KEY}" "DisplayIcon" "$INSTDIR\logo.ico"
-    WriteRegStr SHCTX "${UNINST_KEY}" "Publisher" "Janet-Lang.org"
-    WriteRegStr SHCTX "${UNINST_KEY}" "HelpLink" "${HELPURL}"
-    WriteRegStr SHCTX "${UNINST_KEY}" "URLUpdateInfo" "${HELPURL}"
-    WriteRegStr SHCTX "${UNINST_KEY}" "URLInfoAbout" "${HELPURL}"
-    WriteRegStr SHCTX "${UNINST_KEY}" "DisplayVersion" "${VERSION}"
-    WriteRegDWORD SHCTX "${UNINST_KEY}" "NoModify" 1
-    WriteRegDWORD SHCTX "${UNINST_KEY}" "NoRepair" 1
-    WriteRegDWORD SHCTX "${UNINST_KEY}" "EstimatedSize" 1000
-    # Add uninstall
-    WriteRegStr SHCTX "${UNINST_KEY}" "UninstallString" "$\"$INSTDIR\uninstall.exe$\" /$MultiUser.InstallMode"
-    WriteRegStr SHCTX "${UNINST_KEY}" "QuietUninstallString" "$\"$INSTDIR\uninstall.exe$\" /$MultiUser.InstallMode /S"
-
-sectionEnd
-
-# Uninstaller
-
-function un.onInit
-    !insertmacro MULTIUSER_UNINIT
-functionEnd
-
-section "uninstall"
-
-    # Remove Start Menu launcher
-    delete "$SMPROGRAMS\Janet.lnk"
-
-    # Remove files
-    delete "$INSTDIR\logo.ico"
-    delete "$INSTDIR\README.md"
-    delete "$INSTDIR\LICENSE"
-    rmdir /r "$INSTDIR\Library"
-    rmdir /r "$INSTDIR\bin"
-    rmdir /r "$INSTDIR\C"
-    rmdir /r "$INSTDIR\docs"
-
-    # Remove env vars
-    !insertmacro DelEnv JANET_PATH
-    !insertmacro DelEnv JANET_HEADERPATH
-    !insertmacro DelEnv JANET_LIBPATH
-    !insertmacro DelEnv JANET_BINPATH
-
-    # Unset PATH
-    ${If} $MultiUser.InstallMode == "AllUsers"
-        EnVar::SetHKLM
-    ${Else}
-        EnVar::SetHKCU
-    ${EndIf}
-    EnVar::DeleteValue "PATH" "$INSTDIR\bin"
-    Pop $0
-
-    # make sure windows knows about the change
-    SendMessage ${HWND_BROADCAST} ${WM_WININICHANGE} 0 "STR:Environment" /TIMEOUT=5000
-
-    # Always delete uninstaller as the last action
-    delete "$INSTDIR\uninstall.exe"
-
-    # Remove uninstaller information from the registry
-    DeleteRegKey SHCTX "${UNINST_KEY}"
-
-sectionEnd

janet.1

@@ -13,8 +13,8 @@ janet \- run the Janet language abstract machine
 .BR args ...
 .SH DESCRIPTION
 Janet is a functional and imperative programming language and bytecode interpreter.
-It is a modern lisp, but lists are replaced by other data structures with better utility
-and performance (arrays, tables, structs, tuples). The language also features bridging
+It is a Lisp-like language, but lists are replaced by other data structures
+(arrays, tables, structs, tuples). The language also features bridging
 to native code written in C, meta-programming with macros, and bytecode assembly.
 
 There is a repl for trying out the language, as well as the ability to run script files.
@@ -96,6 +96,10 @@ Delete everything before the cursor on the input line.
 .BR Ctrl\-W
 Delete one word before the cursor.
 
+.TP 16
+.BR Ctrl\-G
+Show documentation for the current symbol under the cursor.
+
 .TP 16
 .BR Alt\-B/Alt\-F
 Move cursor backwards and forwards one word.
@@ -148,6 +152,12 @@ Read raw input from stdin and forgo prompt history and other readline-like featu
 Execute a string of Janet source. Source code is executed in the order it is encountered, so earlier
 arguments are executed before later ones.
 
+.TP
+.BR \-d
+Enable debug mode. On all terminating signals as well the debug signal, this will
+cause the debugger to come up in the REPL. Same as calling (setdyn :debug true) in a
+default repl.
+
 .TP
 .BR \-n
 Disable ANSI colors in the repl. Has no effect if no repl is run.
@@ -165,7 +175,7 @@ after an error. Persistent mode can be good for debugging and testing.
 
 .TP
 .BR \-q
-Quiet output. Don't print a repl prompt or expression results to stdout.
+Hide the logo in the repl.
 
 .TP
 .BR \-k
@@ -184,8 +194,8 @@ Source should be a path to the Janet module to compile, and output should be the
 resulting image. Output should usually end with the .jimage extension.
 
 .TP
-.BR \-l\ path
-Load a Janet file before running a script or repl. Multiple files can be loaded
+.BR \-l\ lib
+Import a Janet module before running a script or repl. Multiple files can be loaded
 in this manner, and exports from each file will be made available to the script
 or repl.
 

jpm.1

@@ -24,6 +24,10 @@ More interesting are the local commands. For more information on jpm usage, see
 
 .SH FLAGS
 
+.TP
+.BR \-\-nocolor
+Disable color in the jpm debug repl.
+
 .TP
 .BR \-\-verbose
 Print detailed messages of what jpm is doing, including compilation commands and other shell commands.
@@ -32,6 +36,12 @@ Print detailed messages of what jpm is doing, including compilation commands and
 .BR \-\-test
 If passed to jpm install, runs tests before installing. Will run tests recursively on dependencies.
 
+.TP
+.BR \-\-offline
+Prevents jpm from going to network to get dependencies - all dependencies should be in the cache or this command will fail.
+Use this flag with the deps and update-pkgs subcommands. This is not a surefire way to prevent a build script from accessing
+the network, for example, a build script that invokes curl will still have network access.
+
 .SH OPTIONS
 
 .TP
@@ -90,19 +100,20 @@ Builds all artifacts specified in the project.janet file in the current director
 be created in the ./build/ directory.
 
 .TP
-.BR install\ [\fBrepo\fR]
+.BR install\ [\fBrepo...\fR]
 
 When run with no arguments, installs all installable artifacts in the current project to
 the current JANET_MODPATH for modules and JANET_BINPATH for executables and scripts. Can also
 take an optional git repository URL and will install all artifacts in that repository instead.
-When run with an argument, install does not need to be run from a jpm project directory.
+When run with an argument, install does not need to be run from a jpm project directory. Will also
+install multiple dependencies in one command.
 
 .TP
-.BR uninstall\ [\fBname\fR]
+.BR uninstall\ [\fBname...\fR]
 Uninstall a project installed with install. uninstall expects the name of the project, not the
 repository url, path to installed file or executable name. The name of the project must be specified
 at the top of the project.janet file in the declare-project form. If no name is given, uninstalls
-the current project if installed.
+the current project if installed. Will also uninstall multiple packages in one command.
 
 .TP
 .BR clean
@@ -128,6 +139,23 @@ date or too large, clear-cache will remove the cache and jpm will rebuild it
 when needed. clear-cache is a global command, so a project.janet is not
 required.
 
+.TP
+.BR list-installed
+List all installed packages in the current syspath.
+
+.TP
+.BR list-pkgs [\fBsearch\fR]
+List all package aliases in the current package listing that contain the given search string.
+If no search string is given, prints the entire listing.
+
+.TP
+.BR clear-manifest
+jpm creates a manifest directory that contains a list of all installed files.
+By deleting this directory, jpm will think that nothing is installed and will
+try reinstalling everything on the jpm deps or jpm load-lockfile commands. Be careful with
+this command, as it may leave extra files on your system and shouldn't be needed
+most of the time in a healthy install.
+
 .TP
 .BR run\ [\fBrule\fR]
 Run a given rule defined in project.janet. Project definitions files (project.janet) usually
@@ -139,6 +167,12 @@ like make. run will run a single rule or build a single file.
 .BR rules
 List all rules that can be run via run. This is useful for exploring rules in the project.
 
+.TP
+.BR rule-tree\ [\fBroot\fR] [\fdepth\fR]
+Show rule dependency tree in a pretty format. Optionally provide a rule to use as the tree
+root, as well as a max depth to print. By default, prints the full tree for all rules. This
+can be quite long, so it is recommended to give a root rule.
+
 .TP
 .BR show-paths
 Show all of the paths used when installing and building artifacts.
@@ -154,6 +188,23 @@ The main function is the entry point of the program and will receive command lin
 as function arguments. The entry file can import other modules, including native C modules, and
 jpm will attempt to include the dependencies into the generated executable.
 
+.TP
+.BR debug-repl
+Load the current project.janet file and start a repl in it's environment. This lets a user better
+debug the project file, as well as run rules manually.
+
+.TP
+.BR make-lockfile\ [\fBfilename\fR]
+Create a lockfile. A lockfile is a record that describes what dependencies were installed at the
+time of the lockfile's creation, including exact versions. A lockfile can then be later used
+to set up that environment on a different machine via load-lockfile. By default, the lockfile
+is created at lockfile.jdn, although any path can be used.
+
+.TP
+.BR load-lockfile\ [\fBfilename\fR]
+Install dependencies from a lockfile previously created with make-lockfile. By default, will look
+for a lockfile at lockfile.jdn, although any path can be used.
+
 .SH ENVIRONMENT
 
 .B JANET_PATH
@@ -203,5 +254,10 @@ The git repository URL that contains a listing of packages. This allows installi
 is mostly a convenience. However, package dependencies can use short names, package listings
 can be used to choose a particular set of dependency versions for a whole project.
 
+.B JANET_GIT
+.RS
+An optional path to a git executable to use to clone git dependencies. By default, uses "git" on the current $PATH. You shouldn't need to set this
+if you have a normal install of git.
+
 .SH AUTHOR
 Written by Calvin Rose <calsrose@gmail.com>

meson.build

@@ -20,7 +20,7 @@
 
 project('janet', 'c',
   default_options : ['c_std=c99', 'b_lundef=false', 'default_library=both'],
-  version : '1.8.1')
+  version : '1.11.1')
 
 # Global settings
 janet_path = join_paths(get_option('prefix'), get_option('libdir'), 'janet')
@@ -59,14 +59,18 @@ conf.set('JANET_NO_DOCSTRINGS', not get_option('docstrings'))
 conf.set('JANET_NO_SOURCEMAPS', not get_option('sourcemaps'))
 conf.set('JANET_NO_ASSEMBLER', not get_option('assembler'))
 conf.set('JANET_NO_PEG', not get_option('peg'))
+conf.set('JANET_NO_NET', not get_option('net'))
 conf.set('JANET_REDUCED_OS', get_option('reduced_os'))
 conf.set('JANET_NO_TYPED_ARRAY', not get_option('typed_array'))
 conf.set('JANET_NO_INT_TYPES', not get_option('int_types'))
-conf.set('JANET_NO_PRF', not get_option('prf'))
+conf.set('JANET_PRF', get_option('prf'))
 conf.set('JANET_RECURSION_GUARD', get_option('recursion_guard'))
 conf.set('JANET_MAX_PROTO_DEPTH', get_option('max_proto_depth'))
 conf.set('JANET_MAX_MACRO_EXPAND', get_option('max_macro_expand'))
 conf.set('JANET_STACK_MAX', get_option('stack_max'))
+conf.set('JANET_NO_UMASK', not get_option('umask'))
+conf.set('JANET_NO_REALPATH', not get_option('realpath'))
+conf.set('JANET_NO_PROCESSES', not get_option('processes'))
 if get_option('os_name') != ''
   conf.set('JANET_OS_NAME', get_option('os_name'))
 endif
@@ -112,6 +116,7 @@ core_src = [
   'src/core/io.c',
   'src/core/marsh.c',
   'src/core/math.c',
+  'src/core/net.c',
   'src/core/os.c',
   'src/core/parse.c',
   'src/core/peg.c',
@@ -167,6 +172,8 @@ janetc = custom_target('janetc',
 libjanet = library('janet', janetc,
   include_directories : incdir,
   dependencies : [m_dep, dl_dep, thread_dep],
+  version: meson.project_version(),
+  soversion: version_parts[0] + '.' + version_parts[1],
   install : true)
 
 # Extra c flags - adding -fvisibility=hidden matches the Makefile and
@@ -217,7 +224,8 @@ test_files = [
   'test/suite5.janet',
   'test/suite6.janet',
   'test/suite7.janet',
-  'test/suite8.janet'
+  'test/suite8.janet',
+  'test/suite9.janet'
 ]
 foreach t : test_files
   test(t, janet_nativeclient, args : files([t]), workdir : meson.current_source_dir())
@@ -237,10 +245,18 @@ pkg.generate(libjanet,
 
 # Installation
 install_man('janet.1')
-install_man('jpm.1')
 install_headers(['src/include/janet.h', jconf], subdir: 'janet')
-janet_binscripts = [
-  'auxbin/jpm'
-]
-install_data(sources : janet_binscripts, install_dir : get_option('bindir'))
 install_data(sources : ['tools/.keep'], install_dir : join_paths(get_option('libdir'), 'janet'))
+if get_option('peg') and not get_option('reduced_os') and get_option('processes')
+  install_man('jpm.1')
+  patched_jpm = custom_target('patched-jpm',
+    input : ['tools/patch-jpm.janet', 'jpm'],
+    install : true,
+    install_dir : get_option('bindir'),
+    build_by_default : true,
+    output : ['jpm'],
+    command : [janet_nativeclient, '@INPUT@', '@OUTPUT@',
+      '--binpath=' + join_paths(get_option('prefix'), get_option('bindir')),
+      '--libpath=' + join_paths(get_option('prefix'), get_option('libdir')),
+      '--headerpath=' + join_paths(get_option('prefix'), get_option('includedir'))])
+endif

meson_options.txt

@@ -10,7 +10,11 @@ option('assembler', type : 'boolean', value : true)
 option('peg', type : 'boolean', value : true)
 option('typed_array', type : 'boolean', value : true)
 option('int_types', type : 'boolean', value : true)
-option('prf', type : 'boolean', value : true)
+option('prf', type : 'boolean', value : false)
+option('net', type : 'boolean', value : true)
+option('processes', type : 'boolean', value : true)
+option('umask', type : 'boolean', value : true)
+option('realpath', type : 'boolean', value : true)
 
 option('recursion_guard', type : 'integer', min : 10, max : 8000, value : 1024)
 option('max_proto_depth', type : 'integer', min : 10, max : 8000, value : 200)

src/boot/system_test.c

@@ -50,5 +50,20 @@ int system_test() {
     assert(janet_equals(janet_cstringv("a string."), janet_cstringv("a string.")));
     assert(janet_equals(janet_csymbolv("sym"), janet_csymbolv("sym")));
 
+    Janet *t1 = janet_tuple_begin(3);
+    t1[0] = janet_wrap_nil();
+    t1[1] = janet_wrap_integer(4);
+    t1[2] = janet_cstringv("hi");
+    Janet tuple1 = janet_wrap_tuple(janet_tuple_end(t1));
+
+    Janet *t2 = janet_tuple_begin(3);
+    t2[0] = janet_wrap_nil();
+    t2[1] = janet_wrap_integer(4);
+    t2[2] = janet_cstringv("hi");
+    Janet tuple2 = janet_wrap_tuple(janet_tuple_end(t2));
+
+    assert(janet_equals(tuple1, tuple2));
+
+
     return 0;
 }

src/boot/table_test.c

@@ -61,5 +61,11 @@ int table_test() {
     assert(janet_equals(janet_table_get(t2, janet_csymbolv("t2key1")), janet_wrap_integer(10)));
     assert(janet_equals(janet_table_get(t2, janet_csymbolv("t2key2")), janet_wrap_integer(100)));
 
+    assert(t2->count == 4);
+    assert(janet_equals(janet_table_remove(t2, janet_csymbolv("t2key1")), janet_wrap_integer(10)));
+    assert(t2->count == 3);
+    assert(janet_equals(janet_table_remove(t2, janet_csymbolv("t2key2")), janet_wrap_integer(100)));
+    assert(t2->count == 2);
+
     return 0;
 }

src/conf/janetconf.h

@@ -27,10 +27,10 @@
 #define JANETCONF_H
 
 #define JANET_VERSION_MAJOR 1
-#define JANET_VERSION_MINOR 8
+#define JANET_VERSION_MINOR 11
 #define JANET_VERSION_PATCH 1
 #define JANET_VERSION_EXTRA ""
-#define JANET_VERSION "1.8.1"
+#define JANET_VERSION "1.11.1"
 
 /* #define JANET_BUILD "local" */
 
@@ -41,18 +41,27 @@
 /* #define JANET_API __attribute__((visibility ("default"))) */
 
 /* These settings should be specified before amalgamation is
- * built. */
+ * built. Any build with these set should be considered non-standard, and
+ * certain Janet libraries should be expected not to work. */
 /* #define JANET_NO_DOCSTRINGS */
 /* #define JANET_NO_SOURCEMAPS */
 /* #define JANET_REDUCED_OS */
-
-/* Other settings */
+/* #define JANET_NO_PROCESSES */
 /* #define JANET_NO_ASSEMBLER */
 /* #define JANET_NO_PEG */
+/* #define JANET_NO_NET */
 /* #define JANET_NO_TYPED_ARRAY */
 /* #define JANET_NO_INT_TYPES */
-/* #define JANET_NO_PRF */
+/* #define JANET_NO_REALPATH */
+/* #define JANET_NO_SYMLINKS */
+/* #define JANET_NO_UMASK */
+
+/* Other settings */
+/* #define JANET_PRF */
+/* #define JANET_NO_UTC_MKTIME */
 /* #define JANET_OUT_OF_MEMORY do { printf("janet out of memory\n"); exit(1); } while (0) */
+/* #define JANET_EXIT(msg) do { printf("C assert failed executing janet: %s\n", msg); exit(1); } while (0) */
+/* #define JANET_TOP_LEVEL_SIGNAL(msg) call_my_function((msg), stderr) */
 /* #define JANET_RECURSION_GUARD 1024 */
 /* #define JANET_MAX_PROTO_DEPTH 200 */
 /* #define JANET_MAX_MACRO_EXPAND 200 */
@@ -60,4 +69,7 @@
 /* #define JANET_OS_NAME my-custom-os */
 /* #define JANET_ARCH_NAME pdp-8 */
 
+/* Main client settings, does not affect library code */
+/* #define JANET_SIMPLE_GETLINE */
+
 #endif /* end of include guard: JANETCONF_H */

src/core/array.c

@@ -270,6 +270,26 @@ static Janet cfun_array_remove(int32_t argc, Janet *argv) {
     return argv[0];
 }
 
+static Janet cfun_array_trim(int32_t argc, Janet *argv) {
+    janet_fixarity(argc, 1);
+    JanetArray *array = janet_getarray(argv, 0);
+    if (array->count) {
+        if (array->count < array->capacity) {
+            Janet *newData = realloc(array->data, array->count * sizeof(Janet));
+            if (NULL == newData) {
+                JANET_OUT_OF_MEMORY;
+            }
+            array->data = newData;
+            array->capacity = array->count;
+        }
+    } else {
+        array->capacity = 0;
+        free(array->data);
+        array->data = NULL;
+    }
+    return argv[0];
+}
+
 static const JanetReg array_cfuns[] = {
     {
         "array/new", cfun_array_new,
@@ -345,6 +365,11 @@ static const JanetReg array_cfuns[] = {
              "By default, n is 1. "
              "Returns the array.")
     },
+    {
+        "array/trim", cfun_array_trim,
+        JDOC("(array/trim arr)\n\n"
+             "Set the backing capacity of an array to its current length. Returns the modified array.")
+    },
     {NULL, NULL, NULL}
 };
 

src/core/asm.c

@@ -53,7 +53,6 @@ struct JanetAssembler {
 
     Janet name;
     JanetTable labels; /* keyword -> bytecode index */
-    JanetTable constants; /* symbol -> constant index */
     JanetTable slots; /* symbol -> slot index */
     JanetTable envs; /* symbol -> environment index */
     JanetTable defs; /* symbol -> funcdefs index */
@@ -113,6 +112,8 @@ static const JanetInstructionDef janet_ops[] = {
     {"movn", JOP_MOVE_NEAR},
     {"mul", JOP_MULTIPLY},
     {"mulim", JOP_MULTIPLY_IMMEDIATE},
+    {"neq", JOP_NOT_EQUALS},
+    {"neqim", JOP_NOT_EQUALS_IMMEDIATE},
     {"next", JOP_NEXT},
     {"noop", JOP_NOOP},
     {"prop", JOP_PROPAGATE},
@@ -172,7 +173,6 @@ static void janet_asm_deinit(JanetAssembler *a) {
     janet_table_deinit(&a->slots);
     janet_table_deinit(&a->labels);
     janet_table_deinit(&a->envs);
-    janet_table_deinit(&a->constants);
     janet_table_deinit(&a->defs);
 }
 
@@ -252,9 +252,6 @@ static int32_t doarg_1(
         case JANET_OAT_ENVIRONMENT:
             c = &a->envs;
             break;
-        case JANET_OAT_CONSTANT:
-            c = &a->constants;
-            break;
         case JANET_OAT_LABEL:
             c = &a->labels;
             break;
@@ -506,7 +503,6 @@ static JanetAssembleResult janet_asm1(JanetAssembler *parent, Janet source, int
     a.defs_capacity = 0;
     a.name = janet_wrap_nil();
     janet_table_init(&a.labels, 0);
-    janet_table_init(&a.constants, 0);
     janet_table_init(&a.slots, 0);
     janet_table_init(&a.envs, 0);
     janet_table_init(&a.defs, 0);
@@ -534,34 +530,34 @@ static JanetAssembleResult janet_asm1(JanetAssembler *parent, Janet source, int
                      "expected struct or table for assembly source");
 
     /* Check for function name */
-    a.name = janet_get1(s, janet_csymbolv("name"));
+    a.name = janet_get1(s, janet_ckeywordv("name"));
     if (!janet_checktype(a.name, JANET_NIL)) {
         def->name = janet_to_string(a.name);
     }
 
     /* Set function arity */
-    x = janet_get1(s, janet_csymbolv("arity"));
+    x = janet_get1(s, janet_ckeywordv("arity"));
     def->arity = janet_checkint(x) ? janet_unwrap_integer(x) : 0;
     janet_asm_assert(&a, def->arity >= 0, "arity must be non-negative");
 
-    x = janet_get1(s, janet_csymbolv("max-arity"));
+    x = janet_get1(s, janet_ckeywordv("max-arity"));
     def->max_arity = janet_checkint(x) ? janet_unwrap_integer(x) : def->arity;
     janet_asm_assert(&a, def->max_arity >= def->arity, "max-arity must be greater than or equal to arity");
 
-    x = janet_get1(s, janet_csymbolv("min-arity"));
+    x = janet_get1(s, janet_ckeywordv("min-arity"));
     def->min_arity = janet_checkint(x) ? janet_unwrap_integer(x) : def->arity;
     janet_asm_assert(&a, def->min_arity <= def->arity, "min-arity must be less than or equal to arity");
 
     /* Check vararg */
-    x = janet_get1(s, janet_csymbolv("vararg"));
+    x = janet_get1(s, janet_ckeywordv("vararg"));
     if (janet_truthy(x)) def->flags |= JANET_FUNCDEF_FLAG_VARARG;
 
     /* Check source */
-    x = janet_get1(s, janet_csymbolv("source"));
+    x = janet_get1(s, janet_ckeywordv("source"));
     if (janet_checktype(x, JANET_STRING)) def->source = janet_unwrap_string(x);
 
     /* Create slot aliases */
-    x = janet_get1(s, janet_csymbolv("slots"));
+    x = janet_get1(s, janet_ckeywordv("slots"));
     if (janet_indexed_view(x, &arr, &count)) {
         for (i = 0; i < count; i++) {
             Janet v = arr[i];
@@ -582,7 +578,7 @@ static JanetAssembleResult janet_asm1(JanetAssembler *parent, Janet source, int
     }
 
     /* Parse constants */
-    x = janet_get1(s, janet_csymbolv("constants"));
+    x = janet_get1(s, janet_ckeywordv("constants"));
     if (janet_indexed_view(x, &arr, &count)) {
         def->constants_length = count;
         def->constants = malloc(sizeof(Janet) * (size_t) count);
@@ -591,25 +587,7 @@ static JanetAssembleResult janet_asm1(JanetAssembler *parent, Janet source, int
         }
         for (i = 0; i < count; i++) {
             Janet ct = arr[i];
-            if (janet_checktype(ct, JANET_TUPLE) &&
-                    janet_tuple_length(janet_unwrap_tuple(ct)) > 1 &&
-                    janet_checktype(janet_unwrap_tuple(ct)[0], JANET_SYMBOL)) {
-                const Janet *t = janet_unwrap_tuple(ct);
-                int32_t tcount = janet_tuple_length(t);
-                const uint8_t *macro = janet_unwrap_symbol(t[0]);
-                if (0 == janet_cstrcmp(macro, "quote")) {
-                    def->constants[i] = t[1];
-                } else if (tcount == 3 &&
-                           janet_checktype(t[1], JANET_SYMBOL) &&
-                           0 == janet_cstrcmp(macro, "def")) {
-                    def->constants[i] = t[2];
-                    janet_table_put(&a.constants, t[1], janet_wrap_integer(i));
-                } else {
-                    janet_asm_errorv(&a, janet_formatc("could not parse constant \"%v\"", ct));
-                }
-            } else {
-                def->constants[i] = ct;
-            }
+            def->constants[i] = ct;
         }
     } else {
         def->constants = NULL;
@@ -617,7 +595,7 @@ static JanetAssembleResult janet_asm1(JanetAssembler *parent, Janet source, int
     }
 
     /* Parse sub funcdefs */
-    x = janet_get1(s, janet_csymbolv("closures"));
+    x = janet_get1(s, janet_ckeywordv("closures"));
     if (janet_indexed_view(x, &arr, &count)) {
         int32_t i;
         for (i = 0; i < count; i++) {
@@ -628,7 +606,7 @@ static JanetAssembleResult janet_asm1(JanetAssembler *parent, Janet source, int
             if (subres.status != JANET_ASSEMBLE_OK) {
                 janet_asm_errorv(&a, subres.error);
             }
-            subname = janet_get1(arr[i], janet_csymbolv("name"));
+            subname = janet_get1(arr[i], janet_ckeywordv("name"));
             if (!janet_checktype(subname, JANET_NIL)) {
                 janet_table_put(&a.defs, subname, janet_wrap_integer(def->defs_length));
             }
@@ -647,7 +625,7 @@ static JanetAssembleResult janet_asm1(JanetAssembler *parent, Janet source, int
     }
 
     /* Parse bytecode and labels */
-    x = janet_get1(s, janet_csymbolv("bytecode"));
+    x = janet_get1(s, janet_ckeywordv("bytecode"));
     if (janet_indexed_view(x, &arr, &count)) {
         /* Do labels and find length */
         int32_t blength = 0;
@@ -703,10 +681,13 @@ static JanetAssembleResult janet_asm1(JanetAssembler *parent, Janet source, int
     a.errindex = -1;
 
     /* Check for source mapping */
-    x = janet_get1(s, janet_csymbolv("sourcemap"));
+    x = janet_get1(s, janet_ckeywordv("sourcemap"));
     if (janet_indexed_view(x, &arr, &count)) {
         janet_asm_assert(&a, count == def->bytecode_length, "sourcemap must have the same length as the bytecode");
         def->sourcemap = malloc(sizeof(JanetSourceMapping) * (size_t) count);
+        if (NULL == def->sourcemap) {
+            JANET_OUT_OF_MEMORY;
+        }
         for (i = 0; i < count; i++) {
             const Janet *tup;
             Janet entry = arr[i];
@@ -730,12 +711,18 @@ static JanetAssembleResult janet_asm1(JanetAssembler *parent, Janet source, int
     /* Set environments */
     def->environments =
         realloc(def->environments, def->environments_length * sizeof(int32_t));
+    if (NULL == def->environments) {
+        JANET_OUT_OF_MEMORY;
+    }
 
     /* Verify the func def */
     if (janet_verify(def)) {
         janet_asm_error(&a, "invalid assembly");
     }
 
+    /* Add final flags */
+    janet_def_addflags(def);
+
     /* Finish everything and return funcdef */
     janet_asm_deinit(&a);
     result.error = NULL;
@@ -858,33 +845,26 @@ Janet janet_disasm(JanetFuncDef *def) {
     JanetArray *bcode = janet_array(def->bytecode_length);
     JanetArray *constants;
     JanetTable *ret = janet_table(10);
-    janet_table_put(ret, janet_csymbolv("arity"), janet_wrap_integer(def->arity));
-    janet_table_put(ret, janet_csymbolv("min-arity"), janet_wrap_integer(def->min_arity));
-    janet_table_put(ret, janet_csymbolv("max-arity"), janet_wrap_integer(def->max_arity));
-    janet_table_put(ret, janet_csymbolv("bytecode"), janet_wrap_array(bcode));
+    janet_table_put(ret, janet_ckeywordv("arity"), janet_wrap_integer(def->arity));
+    janet_table_put(ret, janet_ckeywordv("min-arity"), janet_wrap_integer(def->min_arity));
+    janet_table_put(ret, janet_ckeywordv("max-arity"), janet_wrap_integer(def->max_arity));
+    janet_table_put(ret, janet_ckeywordv("bytecode"), janet_wrap_array(bcode));
     if (NULL != def->source) {
-        janet_table_put(ret, janet_csymbolv("source"), janet_wrap_string(def->source));
+        janet_table_put(ret, janet_ckeywordv("source"), janet_wrap_string(def->source));
     }
     if (def->flags & JANET_FUNCDEF_FLAG_VARARG) {
-        janet_table_put(ret, janet_csymbolv("vararg"), janet_wrap_true());
+        janet_table_put(ret, janet_ckeywordv("vararg"), janet_wrap_true());
     }
     if (NULL != def->name) {
-        janet_table_put(ret, janet_csymbolv("name"), janet_wrap_string(def->name));
+        janet_table_put(ret, janet_ckeywordv("name"), janet_wrap_string(def->name));
     }
 
     /* Add constants */
     if (def->constants_length > 0) {
         constants = janet_array(def->constants_length);
-        janet_table_put(ret, janet_csymbolv("constants"), janet_wrap_array(constants));
+        janet_table_put(ret, janet_ckeywordv("constants"), janet_wrap_array(constants));
         for (i = 0; i < def->constants_length; i++) {
-            Janet src = def->constants[i];
-            Janet dest;
-            if (janet_checktype(src, JANET_TUPLE)) {
-                dest = janet_wrap_tuple(tup2(janet_csymbolv("quote"), src));
-            } else {
-                dest = src;
-            }
-            constants->data[i] = dest;
+            constants->data[i] = def->constants[i];
         }
         constants->count = def->constants_length;
     }
@@ -906,7 +886,7 @@ Janet janet_disasm(JanetFuncDef *def) {
             sourcemap->data[i] = janet_wrap_tuple(janet_tuple_end(t));
         }
         sourcemap->count = def->bytecode_length;
-        janet_table_put(ret, janet_csymbolv("sourcemap"), janet_wrap_array(sourcemap));
+        janet_table_put(ret, janet_ckeywordv("sourcemap"), janet_wrap_array(sourcemap));
     }
 
     /* Add environments */
@@ -916,7 +896,7 @@ Janet janet_disasm(JanetFuncDef *def) {
             envs->data[i] = janet_wrap_integer(def->environments[i]);
         }
         envs->count = def->environments_length;
-        janet_table_put(ret, janet_csymbolv("environments"), janet_wrap_array(envs));
+        janet_table_put(ret, janet_ckeywordv("environments"), janet_wrap_array(envs));
     }
 
     /* Add closures */
@@ -927,11 +907,11 @@ Janet janet_disasm(JanetFuncDef *def) {
             defs->data[i] = janet_disasm(def->defs[i]);
         }
         defs->count = def->defs_length;
-        janet_table_put(ret, janet_csymbolv("defs"), janet_wrap_array(defs));
+        janet_table_put(ret, janet_ckeywordv("defs"), janet_wrap_array(defs));
     }
 
     /* Add slotcount */
-    janet_table_put(ret, janet_csymbolv("slotcount"), janet_wrap_integer(def->slotcount));
+    janet_table_put(ret, janet_ckeywordv("slotcount"), janet_wrap_integer(def->slotcount));
 
     return janet_wrap_struct(janet_table_to_struct(ret));
 }
@@ -958,7 +938,7 @@ static const JanetReg asm_cfuns[] = {
         "asm", cfun_asm,
         JDOC("(asm assembly)\n\n"
              "Returns a new function that is the compiled result of the assembly.\n"
-             "The syntax for the assembly can be found on the janet wiki. Will throw an\n"
+             "The syntax for the assembly can be found on the Janet website. Will throw an\n"
              "error on invalid assembly.")
     },
     {

src/core/buffer.c

@@ -197,6 +197,26 @@ static Janet cfun_buffer_fill(int32_t argc, Janet *argv) {
     return argv[0];
 }
 
+static Janet cfun_buffer_trim(int32_t argc, Janet *argv) {
+    janet_fixarity(argc, 1);
+    JanetBuffer *buffer = janet_getbuffer(argv, 0);
+    if (buffer->count) {
+        if (buffer->count < buffer->capacity) {
+            uint8_t *newData = realloc(buffer->data, buffer->count);
+            if (NULL == newData) {
+                JANET_OUT_OF_MEMORY;
+            }
+            buffer->data = newData;
+            buffer->capacity = buffer->count;
+        }
+    } else {
+        buffer->capacity = 0;
+        free(buffer->data);
+        buffer->data = NULL;
+    }
+    return argv[0];
+}
+
 static Janet cfun_buffer_u8(int32_t argc, Janet *argv) {
     int32_t i;
     janet_arity(argc, 1, -1);
@@ -379,6 +399,12 @@ static const JanetReg buffer_cfuns[] = {
              "Fill up a buffer with bytes, defaulting to 0s. Does not change the buffer's length. "
              "Returns the modified buffer.")
     },
+    {
+        "buffer/trim", cfun_buffer_trim,
+        JDOC("(buffer/trim buffer)\n\n"
+             "Set the backing capacity of the buffer to the current length of the buffer. Returns the "
+             "modified buffer.")
+    },
     {
         "buffer/push-byte", cfun_buffer_u8,
         JDOC("(buffer/push-byte buffer x)\n\n"

src/core/bytecode.c

@@ -101,10 +101,12 @@ enum JanetInstructionType janet_instructions[JOP_INSTRUCTION_COUNT] = {
     JINT_SSS, /* JOP_GREATER_THAN_EQUAL */
     JINT_SSS, /* JOP_LESS_THAN_EQUAL */
     JINT_SSS, /* JOP_NEXT */
+    JINT_SSS, /* JOP_NOT_EQUALS, */
+    JINT_SSI, /* JOP_NOT_EQUALS_IMMEDIATE, */
 };
 
 /* Verify some bytecode */
-int32_t janet_verify(JanetFuncDef *def) {
+int janet_verify(JanetFuncDef *def) {
     int vargs = !!(def->flags & JANET_FUNCDEF_FLAG_VARARG);
     int32_t i;
     int32_t maxslot = def->arity + vargs;

src/core/capi.c

@@ -27,6 +27,29 @@
 #include "fiber.h"
 #endif
 
+#ifndef JANET_SINGLE_THREADED
+#ifndef JANET_WINDOWS
+#include <pthread.h>
+#else
+#include <windows.h>
+#endif
+#endif
+
+JANET_NO_RETURN static void janet_top_level_signal(const char *msg) {
+#ifdef JANET_TOP_LEVEL_SIGNAL
+    JANET_TOP_LEVEL_SIGNAL(msg);
+#else
+    fputs(msg, stdout);
+# ifdef JANET_SINGLE_THREADED
+    exit(-1);
+# elif defined(JANET_WINDOWS)
+    ExitThread(-1);
+# else
+    pthread_exit(NULL);
+# endif
+#endif
+}
+
 void janet_signalv(JanetSignal sig, Janet message) {
     if (janet_vm_return_reg != NULL) {
         *janet_vm_return_reg = message;
@@ -37,8 +60,8 @@ void janet_signalv(JanetSignal sig, Janet message) {
         longjmp(*janet_vm_jmp_buf, sig);
 #endif
     } else {
-        fputs((const char *)janet_formatc("janet top level signal - %v\n", message), stdout);
-        exit(1);
+        const char *str = (const char *)janet_formatc("janet top level signal - %v\n", message);
+        janet_top_level_signal(str);
     }
 }
 
@@ -54,7 +77,7 @@ void janet_panicf(const char *format, ...) {
     while (format[len]) len++;
     janet_buffer_init(&buffer, len);
     va_start(args, format);
-    janet_formatb(&buffer, format, args);
+    janet_formatbv(&buffer, format, args);
     va_end(args);
     ret = janet_string(buffer.data, buffer.count);
     janet_buffer_deinit(&buffer);
@@ -235,18 +258,20 @@ size_t janet_getsize(const Janet *argv, int32_t n) {
 
 int32_t janet_gethalfrange(const Janet *argv, int32_t n, int32_t length, const char *which) {
     int32_t raw = janet_getinteger(argv, n);
-    if (raw < 0) raw += length + 1;
-    if (raw < 0 || raw > length)
-        janet_panicf("%s index %d out of range [0,%d]", which, raw, length);
-    return raw;
+    int32_t not_raw = raw;
+    if (not_raw < 0) not_raw += length + 1;
+    if (not_raw < 0 || not_raw > length)
+        janet_panicf("%s index %d out of range [%d,%d]", which, raw, -length - 1, length);
+    return not_raw;
 }
 
 int32_t janet_getargindex(const Janet *argv, int32_t n, int32_t length, const char *which) {
     int32_t raw = janet_getinteger(argv, n);
-    if (raw < 0) raw += length;
-    if (raw < 0 || raw > length)
-        janet_panicf("%s index %d out of range [0,%d)", which, raw, length);
-    return raw;
+    int32_t not_raw = raw;
+    if (not_raw < 0) not_raw += length;
+    if (not_raw < 0 || not_raw > length)
+        janet_panicf("%s index %d out of range [%d,%d)", which, raw, -length, length);
+    return not_raw;
 }
 
 JanetView janet_getindexed(const Janet *argv, int32_t n) {
@@ -314,7 +339,10 @@ JanetRange janet_getslice(int32_t argc, const Janet *argv) {
 }
 
 Janet janet_dyn(const char *name) {
-    if (!janet_vm_fiber) return janet_wrap_nil();
+    if (!janet_vm_fiber) {
+        if (!janet_vm_top_dyns) return janet_wrap_nil();
+        return janet_table_get(janet_vm_top_dyns, janet_ckeywordv(name));
+    }
     if (janet_vm_fiber->env) {
         return janet_table_get(janet_vm_fiber->env, janet_ckeywordv(name));
     } else {
@@ -323,11 +351,15 @@ Janet janet_dyn(const char *name) {
 }
 
 void janet_setdyn(const char *name, Janet value) {
-    if (!janet_vm_fiber) return;
-    if (!janet_vm_fiber->env) {
-        janet_vm_fiber->env = janet_table(1);
+    if (!janet_vm_fiber) {
+        if (!janet_vm_top_dyns) janet_vm_top_dyns = janet_table(10);
+        janet_table_put(janet_vm_top_dyns, janet_ckeywordv(name), value);
+    } else {
+        if (!janet_vm_fiber->env) {
+            janet_vm_fiber->env = janet_table(1);
+        }
+        janet_table_put(janet_vm_fiber->env, janet_ckeywordv(name), value);
     }
-    janet_table_put(janet_vm_fiber->env, janet_ckeywordv(name), value);
 }
 
 uint64_t janet_getflags(const Janet *argv, int32_t n, const char *flags) {

src/core/cfuns.c

@@ -33,6 +33,11 @@ static int arity1or2(JanetFopts opts, JanetSlot *args) {
     int32_t arity = janet_v_count(args);
     return arity == 1 || arity == 2;
 }
+static int arity2or3(JanetFopts opts, JanetSlot *args) {
+    (void) opts;
+    int32_t arity = janet_v_count(args);
+    return arity == 2 || arity == 3;
+}
 static int fixarity1(JanetFopts opts, JanetSlot *args) {
     (void) opts;
     return janet_v_count(args) == 1;
@@ -90,34 +95,67 @@ static JanetSlot opfunction(
     return t;
 }
 
+/* Check if a value can be coerced to an immediate value */
+static int can_be_imm(Janet x, int8_t *out) {
+    if (!janet_checkint(x)) return 0;
+    int32_t integer = janet_unwrap_integer(x);
+    if (integer > 127 || integer < -127) return 0;
+    *out = (int8_t) integer;
+    return 1;
+}
+
+/* Check if a slot can be coerced to an immediate value */
+static int can_slot_be_imm(JanetSlot s, int8_t *out) {
+    if (!(s.flags & JANET_SLOT_CONSTANT)) return 0;
+    return can_be_imm(s.constant, out);
+}
+
 /* Emit a series of instructions instead of a function call to a math op */
 static JanetSlot opreduce(
     JanetFopts opts,
     JanetSlot *args,
     int op,
+    int opim,
     Janet nullary) {
     JanetCompiler *c = opts.compiler;
     int32_t i, len;
+    int8_t imm = 0;
+    int neg = opim < 0;
+    if (opim < 0) opim = -opim;
     len = janet_v_count(args);
     JanetSlot t;
     if (len == 0) {
         return janetc_cslot(nullary);
     } else if (len == 1) {
         t = janetc_gettarget(opts);
-        janetc_emit_sss(c, op, t, janetc_cslot(nullary), args[0], 1);
+        /* Special case subtract to be times -1 */
+        if (op == JOP_SUBTRACT) {
+            janetc_emit_ssi(c, JOP_MULTIPLY_IMMEDIATE, t, args[0], -1, 1);
+        } else {
+            janetc_emit_sss(c, op, t, janetc_cslot(nullary), args[0], 1);
+        }
         return t;
     }
     t = janetc_gettarget(opts);
-    janetc_emit_sss(c, op, t, args[0], args[1], 1);
-    for (i = 2; i < len; i++)
-        janetc_emit_sss(c, op, t, t, args[i], 1);
+    if (opim && can_slot_be_imm(args[1], &imm)) {
+        janetc_emit_ssi(c, opim, t, args[0], neg ? -imm : imm, 1);
+    } else {
+        janetc_emit_sss(c, op, t, args[0], args[1], 1);
+    }
+    for (i = 2; i < len; i++) {
+        if (opim && can_slot_be_imm(args[i], &imm)) {
+            janetc_emit_ssi(c, opim, t, t, neg ? -imm : imm, 1);
+        } else {
+            janetc_emit_sss(c, op, t, t, args[i], 1);
+        }
+    }
     return t;
 }
 
 /* Function optimizers */
 
 static JanetSlot do_propagate(JanetFopts opts, JanetSlot *args) {
-    return opreduce(opts, args, JOP_PROPAGATE, janet_wrap_nil());
+    return opreduce(opts, args, JOP_PROPAGATE, 0, janet_wrap_nil());
 }
 static JanetSlot do_error(JanetFopts opts, JanetSlot *args) {
     janetc_emit_s(opts.compiler, JOP_ERROR, args[0], 0);
@@ -134,19 +172,40 @@ static JanetSlot do_debug(JanetFopts opts, JanetSlot *args) {
     return t;
 }
 static JanetSlot do_in(JanetFopts opts, JanetSlot *args) {
-    return opreduce(opts, args, JOP_IN, janet_wrap_nil());
+    return opreduce(opts, args, JOP_IN, 0, janet_wrap_nil());
 }
 static JanetSlot do_get(JanetFopts opts, JanetSlot *args) {
-    return opreduce(opts, args, JOP_GET, janet_wrap_nil());
+    if (janet_v_count(args) == 3) {
+        JanetCompiler *c = opts.compiler;
+        JanetSlot t = janetc_gettarget(opts);
+        int target_is_default = janetc_sequal(t, args[2]);
+        JanetSlot dflt_slot = args[2];
+        if (target_is_default) {
+            dflt_slot = janetc_farslot(c);
+            janetc_copy(c, dflt_slot, t);
+        }
+        janetc_emit_sss(c, JOP_GET, t, args[0], args[1], 1);
+        int32_t label = janetc_emit_si(c, JOP_JUMP_IF_NOT_NIL, t, 0, 0);
+        janetc_copy(c, t, dflt_slot);
+        if (target_is_default) janetc_freeslot(c, dflt_slot);
+        int32_t current = janet_v_count(c->buffer);
+        c->buffer[label] |= (current - label) << 16;
+        return t;
+    } else {
+        return opreduce(opts, args, JOP_GET, 0, janet_wrap_nil());
+    }
 }
 static JanetSlot do_next(JanetFopts opts, JanetSlot *args) {
     return opfunction(opts, args, JOP_NEXT, janet_wrap_nil());
 }
 static JanetSlot do_modulo(JanetFopts opts, JanetSlot *args) {
-    return opreduce(opts, args, JOP_MODULO, janet_wrap_nil());
+    return opreduce(opts, args, JOP_MODULO, 0, janet_wrap_nil());
 }
 static JanetSlot do_remainder(JanetFopts opts, JanetSlot *args) {
-    return opreduce(opts, args, JOP_REMAINDER, janet_wrap_nil());
+    return opreduce(opts, args, JOP_REMAINDER, 0, janet_wrap_nil());
+}
+static JanetSlot do_cmp(JanetFopts opts, JanetSlot *args) {
+    return opreduce(opts, args, JOP_COMPARE, 0, janet_wrap_nil());
 }
 static JanetSlot do_put(JanetFopts opts, JanetSlot *args) {
     if (opts.flags & JANET_FOPTS_DROP) {
@@ -200,34 +259,34 @@ static JanetSlot do_apply(JanetFopts opts, JanetSlot *args) {
 /* Variadic operators specialization */
 
 static JanetSlot do_add(JanetFopts opts, JanetSlot *args) {
-    return opreduce(opts, args, JOP_ADD, janet_wrap_integer(0));
+    return opreduce(opts, args, JOP_ADD, JOP_ADD_IMMEDIATE, janet_wrap_integer(0));
 }
 static JanetSlot do_sub(JanetFopts opts, JanetSlot *args) {
-    return opreduce(opts, args, JOP_SUBTRACT, janet_wrap_integer(0));
+    return opreduce(opts, args, JOP_SUBTRACT, -JOP_ADD_IMMEDIATE, janet_wrap_integer(0));
 }
 static JanetSlot do_mul(JanetFopts opts, JanetSlot *args) {
-    return opreduce(opts, args, JOP_MULTIPLY, janet_wrap_integer(1));
+    return opreduce(opts, args, JOP_MULTIPLY, JOP_MULTIPLY_IMMEDIATE, janet_wrap_integer(1));
 }
 static JanetSlot do_div(JanetFopts opts, JanetSlot *args) {
-    return opreduce(opts, args, JOP_DIVIDE, janet_wrap_integer(1));
+    return opreduce(opts, args, JOP_DIVIDE, JOP_DIVIDE_IMMEDIATE, janet_wrap_integer(1));
 }
 static JanetSlot do_band(JanetFopts opts, JanetSlot *args) {
-    return opreduce(opts, args, JOP_BAND, janet_wrap_integer(-1));
+    return opreduce(opts, args, JOP_BAND, 0, janet_wrap_integer(-1));
 }
 static JanetSlot do_bor(JanetFopts opts, JanetSlot *args) {
-    return opreduce(opts, args, JOP_BOR, janet_wrap_integer(0));
+    return opreduce(opts, args, JOP_BOR, 0, janet_wrap_integer(0));
 }
 static JanetSlot do_bxor(JanetFopts opts, JanetSlot *args) {
-    return opreduce(opts, args, JOP_BXOR, janet_wrap_integer(0));
+    return opreduce(opts, args, JOP_BXOR, 0, janet_wrap_integer(0));
 }
 static JanetSlot do_lshift(JanetFopts opts, JanetSlot *args) {
-    return opreduce(opts, args, JOP_SHIFT_LEFT, janet_wrap_integer(1));
+    return opreduce(opts, args, JOP_SHIFT_LEFT, JOP_SHIFT_LEFT_IMMEDIATE, janet_wrap_integer(1));
 }
 static JanetSlot do_rshift(JanetFopts opts, JanetSlot *args) {
-    return opreduce(opts, args, JOP_SHIFT_RIGHT, janet_wrap_integer(1));
+    return opreduce(opts, args, JOP_SHIFT_RIGHT, JOP_SHIFT_RIGHT_IMMEDIATE, janet_wrap_integer(1));
 }
 static JanetSlot do_rshiftu(JanetFopts opts, JanetSlot *args) {
-    return opreduce(opts, args, JOP_SHIFT_RIGHT, janet_wrap_integer(1));
+    return opreduce(opts, args, JOP_SHIFT_RIGHT_UNSIGNED, JOP_SHIFT_RIGHT_UNSIGNED_IMMEDIATE, janet_wrap_integer(1));
 }
 static JanetSlot do_bnot(JanetFopts opts, JanetSlot *args) {
     return genericSS(opts, JOP_BNOT, args[0]);
@@ -238,9 +297,11 @@ static JanetSlot compreduce(
     JanetFopts opts,
     JanetSlot *args,
     int op,
+    int opim,
     int invert) {
     JanetCompiler *c = opts.compiler;
     int32_t i, len;
+    int8_t imm = 0;
     len = janet_v_count(args);
     int32_t *labels = NULL;
     JanetSlot t;
@@ -251,19 +312,17 @@ static JanetSlot compreduce(
     }
     t = janetc_gettarget(opts);
     for (i = 1; i < len; i++) {
-        janetc_emit_sss(c, op, t, args[i - 1], args[i], 1);
+        if (opim && can_slot_be_imm(args[i], &imm)) {
+            janetc_emit_ssi(c, opim, t, args[i - 1], imm, 1);
+        } else {
+            janetc_emit_sss(c, op, t, args[i - 1], args[i], 1);
+        }
         if (i != (len - 1)) {
-            int32_t label = janetc_emit_si(c, JOP_JUMP_IF_NOT, t, 0, 1);
+            int32_t label = janetc_emit_si(c, invert ? JOP_JUMP_IF : JOP_JUMP_IF_NOT, t, 0, 1);
             janet_v_push(labels, label);
         }
     }
     int32_t end = janet_v_count(c->buffer);
-    if (invert) {
-        janetc_emit_si(c, JOP_JUMP_IF, t, 3, 0);
-        janetc_emit_s(c, JOP_LOAD_TRUE, t, 1);
-        janetc_emit(c, JOP_JUMP | (2 << 8));
-        janetc_emit_s(c, JOP_LOAD_FALSE, t, 1);
-    }
     for (i = 0; i < janet_v_count(labels); i++) {
         int32_t label = labels[i];
         c->buffer[label] |= ((end - label) << 16);
@@ -273,22 +332,22 @@ static JanetSlot compreduce(
 }
 
 static JanetSlot do_gt(JanetFopts opts, JanetSlot *args) {
-    return compreduce(opts, args, JOP_GREATER_THAN, 0);
+    return compreduce(opts, args, JOP_GREATER_THAN, JOP_GREATER_THAN_IMMEDIATE, 0);
 }
 static JanetSlot do_lt(JanetFopts opts, JanetSlot *args) {
-    return compreduce(opts, args, JOP_LESS_THAN, 0);
+    return compreduce(opts, args, JOP_LESS_THAN, JOP_LESS_THAN_IMMEDIATE, 0);
 }
 static JanetSlot do_gte(JanetFopts opts, JanetSlot *args) {
-    return compreduce(opts, args, JOP_GREATER_THAN_EQUAL, 0);
+    return compreduce(opts, args, JOP_GREATER_THAN_EQUAL, 0, 0);
 }
 static JanetSlot do_lte(JanetFopts opts, JanetSlot *args) {
-    return compreduce(opts, args, JOP_LESS_THAN_EQUAL, 0);
+    return compreduce(opts, args, JOP_LESS_THAN_EQUAL, 0, 0);
 }
 static JanetSlot do_eq(JanetFopts opts, JanetSlot *args) {
-    return compreduce(opts, args, JOP_EQUALS, 0);
+    return compreduce(opts, args, JOP_EQUALS, JOP_EQUALS_IMMEDIATE, 0);
 }
 static JanetSlot do_neq(JanetFopts opts, JanetSlot *args) {
-    return compreduce(opts, args, JOP_EQUALS, 1);
+    return compreduce(opts, args, JOP_NOT_EQUALS, JOP_NOT_EQUALS_IMMEDIATE, 1);
 }
 
 /* Arranged by tag */
@@ -319,10 +378,11 @@ static const JanetFunOptimizer optimizers[] = {
     {NULL, do_eq},
     {NULL, do_neq},
     {fixarity2, do_propagate},
-    {fixarity2, do_get},
+    {arity2or3, do_get},
     {arity1or2, do_next},
     {fixarity2, do_modulo},
     {fixarity2, do_remainder},
+    {fixarity2, do_cmp},
 };
 
 const JanetFunOptimizer *janetc_funopt(uint32_t flags) {

src/core/compile.c

@@ -596,8 +596,11 @@ static int macroexpand1(
     /* Set env */
     fiberp->env = c->env;
     int lock = janet_gclock();
+    Janet mf_kw = janet_ckeywordv("macro-form");
+    janet_table_put(c->env, mf_kw, x);
     Janet tempOut;
     JanetSignal status = janet_continue(fiberp, janet_wrap_nil(), &tempOut);
+    janet_table_put(c->env, mf_kw, janet_wrap_nil());
     janet_gcunlock(lock);
     if (status != JANET_SIGNAL_OK) {
         const uint8_t *es = janet_formatc("(macro) %V", tempOut);
@@ -695,7 +698,32 @@ JanetSlot janetc_value(JanetFopts opts, Janet x) {
     return ret;
 }
 
+/* Add function flags to janet functions */
+void janet_def_addflags(JanetFuncDef *def) {
+    int32_t set_flags = 0;
+    int32_t unset_flags = 0;
+    /* pos checks */
+    if (def->name)            set_flags |= JANET_FUNCDEF_FLAG_HASNAME;
+    if (def->source)          set_flags |= JANET_FUNCDEF_FLAG_HASSOURCE;
+    if (def->defs)            set_flags |= JANET_FUNCDEF_FLAG_HASDEFS;
+    if (def->environments)    set_flags |= JANET_FUNCDEF_FLAG_HASENVS;
+    if (def->sourcemap)       set_flags |= JANET_FUNCDEF_FLAG_HASSOURCEMAP;
+    if (def->closure_bitset)  set_flags |= JANET_FUNCDEF_FLAG_HASCLOBITSET;
+    /* negative checks */
+    if (!def->name)           unset_flags |= JANET_FUNCDEF_FLAG_HASNAME;
+    if (!def->source)         unset_flags |= JANET_FUNCDEF_FLAG_HASSOURCE;
+    if (!def->defs)           unset_flags |= JANET_FUNCDEF_FLAG_HASDEFS;
+    if (!def->environments)   unset_flags |= JANET_FUNCDEF_FLAG_HASENVS;
+    if (!def->sourcemap)      unset_flags |= JANET_FUNCDEF_FLAG_HASSOURCEMAP;
+    if (!def->closure_bitset) unset_flags |= JANET_FUNCDEF_FLAG_HASCLOBITSET;
+    /* Update flags */
+    def->flags |= set_flags;
+    def->flags &= ~unset_flags;
+}
+
 /* Compile a funcdef */
+/* Once the various other settings of the FuncDef have been tweaked,
+ * call janet_def_addflags to set the proper flags for the funcdef */
 JanetFuncDef *janetc_pop_funcdef(JanetCompiler *c) {
     JanetScope *scope = c->scope;
     JanetFuncDef *def = janet_funcdef_alloc();
@@ -747,8 +775,10 @@ JanetFuncDef *janetc_pop_funcdef(JanetCompiler *c) {
     /* Copy upvalue bitset */
     if (scope->ua.count) {
         /* Number of u32s we need to create a bitmask for all slots */
-        int32_t numchunks = (def->slotcount + 31) >> 5;
-        uint32_t *chunks = malloc(sizeof(uint32_t) * numchunks);
+        int32_t slotchunks = (def->slotcount + 31) >> 5;
+        /* numchunks is min of slotchunks and scope->ua.count */
+        int32_t numchunks = slotchunks > scope->ua.count ? scope->ua.count : slotchunks;
+        uint32_t *chunks = calloc(sizeof(uint32_t), slotchunks);
         if (NULL == chunks) {
             JANET_OUT_OF_MEMORY;
         }
@@ -756,7 +786,6 @@ JanetFuncDef *janetc_pop_funcdef(JanetCompiler *c) {
         /* Register allocator preallocates some registers [240-255, high 16 bits of chunk index 7], we can ignore those. */
         if (scope->ua.count > 7) chunks[7] &= 0xFFFFU;
         def->closure_bitset = chunks;
-        def->flags |= JANET_FUNCDEF_FLAG_HASCLOBITSET;
     }
 
     /* Pop the scope */
@@ -813,6 +842,7 @@ JanetCompileResult janet_compile(Janet source, JanetTable *env, const uint8_t *w
     if (c.result.status == JANET_COMPILE_OK) {
         JanetFuncDef *def = janetc_pop_funcdef(&c);
         def->name = janet_cstring("_thunk");
+        janet_def_addflags(def);
         c.result.funcdef = def;
     } else {
         c.result.error_mapping = c.current_mapping;
@@ -855,10 +885,10 @@ static const JanetReg compile_cfuns[] = {
     {
         "compile", cfun,
         JDOC("(compile ast &opt env source)\n\n"
-             "Compiles an Abstract Syntax Tree (ast) into a janet function. "
+             "Compiles an Abstract Syntax Tree (ast) into a function. "
              "Pair the compile function with parsing functionality to implement "
-             "eval. Returns a janet function and does not modify ast. Throws an "
-             "error if the ast cannot be compiled.")
+             "eval. Returns a new function and does not modify ast. Returns an error "
+             "struct with keys :line, :column, and :error if compilation fails.")
     },
     {NULL, NULL, NULL}
 };

src/core/compile.h

@@ -60,6 +60,7 @@
 #define JANET_FUN_NEXT 28
 #define JANET_FUN_MODULO 29
 #define JANET_FUN_REMAINDER 30
+#define JANET_FUN_CMP 31
 
 /* Compiler typedefs */
 typedef struct JanetCompiler JanetCompiler;

src/core/corelib.c

@@ -63,10 +63,29 @@ typedef void *Clib;
 #define error_clib() dlerror()
 #endif
 
+static char *get_processed_name(const char *name) {
+    if (name[0] == '.') return (char *) name;
+    const char *c;
+    for (c = name; *c; c++) {
+        if (*c == '/') return (char *) name;
+    }
+    size_t l = (size_t)(c - name);
+    char *ret = malloc(l + 3);
+    if (NULL == ret) {
+        JANET_OUT_OF_MEMORY;
+    }
+    ret[0] = '.';
+    ret[1] = '/';
+    memcpy(ret + 2, name, l + 1);
+    return ret;
+}
+
 JanetModule janet_native(const char *name, const uint8_t **error) {
-    Clib lib = load_clib(name);
+    char *processed_name = get_processed_name(name);
+    Clib lib = load_clib(processed_name);
     JanetModule init;
     JanetModconf getter;
+    if (name != processed_name) free(processed_name);
     if (!lib) {
         *error = janet_cstring(error_clib());
         return NULL;
@@ -404,9 +423,11 @@ static Janet janet_core_gcsetinterval(int32_t argc, Janet *argv) {
     janet_fixarity(argc, 1);
     size_t s = janet_getsize(argv, 0);
     /* limit interval to 48 bits */
-    if (s > 0xFFFFFFFFFFFFUl) {
+#ifdef JANET_64
+    if (s >> 48) {
         janet_panic("interval too large");
     }
+#endif
     janet_vm_gc_interval = s;
     return janet_wrap_nil();
 }
@@ -435,7 +456,7 @@ static Janet janet_core_hash(int32_t argc, Janet *argv) {
 static Janet janet_core_getline(int32_t argc, Janet *argv) {
     FILE *in = janet_dynfile("in", stdin);
     FILE *out = janet_dynfile("out", stdout);
-    janet_arity(argc, 0, 2);
+    janet_arity(argc, 0, 3);
     JanetBuffer *buf = (argc >= 2) ? janet_getbuffer(argv, 1) : janet_buffer(10);
     if (argc >= 1) {
         const char *prompt = (const char *) janet_getstring(argv, 0);
@@ -500,10 +521,15 @@ static Janet janet_core_signal(int32_t argc, Janet *argv) {
         sig = JANET_SIGNAL_USER0 + s;
     } else {
         JanetKeyword kw = janet_getkeyword(argv, 0);
-        if (!janet_cstrcmp(kw, "yield")) sig = JANET_SIGNAL_YIELD;
-        if (!janet_cstrcmp(kw, "error")) sig = JANET_SIGNAL_ERROR;
-        if (!janet_cstrcmp(kw, "debug")) sig = JANET_SIGNAL_DEBUG;
-        janet_panicf("unknown signal, expected :yield, :error, or :debug, got %v", argv[0]);
+        if (!janet_cstrcmp(kw, "yield")) {
+            sig = JANET_SIGNAL_YIELD;
+        } else if (!janet_cstrcmp(kw, "error")) {
+            sig = JANET_SIGNAL_ERROR;
+        } else if (!janet_cstrcmp(kw, "debug")) {
+            sig = JANET_SIGNAL_DEBUG;
+        } else {
+            janet_panicf("unknown signal, expected :yield, :error, or :debug, got %v", argv[0]);
+        }
     }
     Janet payload = argc == 2 ? argv[1] : janet_wrap_nil();
     janet_signalv(sig, payload);
@@ -638,15 +664,15 @@ static const JanetReg corelib_cfuns[] = {
     {
         "hash", janet_core_hash,
         JDOC("(hash value)\n\n"
-             "Gets a hash value for any janet value. The hash is an integer can be used "
-             "as a cheap hash function for all janet objects. If two values are strictly equal, "
+             "Gets a hash for any value. The hash is an integer can be used "
+             "as a cheap hash function for all values. If two values are strictly equal, "
              "then they will have the same hash value.")
     },
     {
         "getline", janet_core_getline,
         JDOC("(getline &opt prompt buf env)\n\n"
              "Reads a line of input into a buffer, including the newline character, using a prompt. "
-             "An optional environment table can be provided for autocomplete. "
+             "An optional environment table can be provided for auto-complete. "
              "Returns the modified buffer. "
              "Use this function to implement a simple interface for a terminal program.")
     },
@@ -680,9 +706,9 @@ static const JanetReg corelib_cfuns[] = {
              "\t:all:\tthe value of path verbatim\n"
              "\t:cur:\tthe current file, or (dyn :current-file)\n"
              "\t:dir:\tthe directory containing the current file\n"
-             "\t:name:\tthe filename component of path, with extenion if given\n"
+             "\t:name:\tthe name component of path, with extension if given\n"
              "\t:native:\tthe extension used to load natives, .so or .dll\n"
-             "\t:sys:\tthe system path, or (syn :syspath)")
+             "\t:sys:\tthe system path, or (dyn :syspath)")
     },
     {
         "int?", janet_core_check_int,
@@ -697,7 +723,7 @@ static const JanetReg corelib_cfuns[] = {
     {
         "slice", janet_core_slice,
         JDOC("(slice x &opt start end)\n\n"
-             "Extract a sub-range of an indexed data strutrue or byte sequence.")
+             "Extract a sub-range of an indexed data structure or byte sequence.")
     },
     {
         "signal", janet_core_signal,
@@ -734,10 +760,11 @@ static void janet_quick_asm(
         JANET_OUT_OF_MEMORY;
     }
     memcpy(def->bytecode, bytecode, bytecode_size);
+    janet_def_addflags(def);
     janet_def(env, name, janet_wrap_function(janet_thunk(def)), doc);
 }
 
-/* Macros for easier inline janet assembly */
+/* Macros for easier inline assembly */
 #define SSS(op, a, b, c) ((op) | ((a) << 8) | ((b) << 16) | ((c) << 24))
 #define SS(op, a, b) ((op) | ((a) << 8) | ((b) << 16))
 #define SSI(op, a, b, I) ((op) | ((a) << 8) | ((b) << 16) | ((uint32_t)(I) << 24))
@@ -963,6 +990,10 @@ static const uint32_t remainder_asm[] = {
     JOP_REMAINDER | (1 << 24),
     JOP_RETURN
 };
+static const uint32_t cmp_asm[] = {
+    JOP_COMPARE | (1 << 24),
+    JOP_RETURN
+};
 #endif /* ifdef JANET_BOOTSTRAP */
 
 /*
@@ -999,6 +1030,9 @@ static void janet_load_libs(JanetTable *env) {
 #ifdef JANET_THREADS
     janet_lib_thread(env);
 #endif
+#ifdef JANET_NET
+    janet_lib_net(env);
+#endif
 }
 
 #ifdef JANET_BOOTSTRAP
@@ -1013,10 +1047,15 @@ JanetTable *janet_core_env(JanetTable *replacements) {
                     "%", 2, 2, 2, 2, remainder_asm, sizeof(remainder_asm),
                     JDOC("(% dividend divisor)\n\n"
                          "Returns the remainder of dividend / divisor."));
+    janet_quick_asm(env, JANET_FUN_CMP,
+                    "cmp", 2, 2, 2, 2, cmp_asm, sizeof(cmp_asm),
+                    JDOC("(cmp x y)\n\n"
+                         "Returns -1 if x is strictly less than y, 1 if y is strictly greater "
+                         "than x, and 0 otherwise. To return 0, x and y must be the exact same type."));
     janet_quick_asm(env, JANET_FUN_NEXT,
                     "next", 2, 1, 2, 2, next_asm, sizeof(next_asm),
                     JDOC("(next ds &opt key)\n\n"
-                         "Gets the next key in a datastructure. Can be used to iterate through "
+                         "Gets the next key in a data structure. Can be used to iterate through "
                          "the keys of a data structure in an unspecified order. Keys are guaranteed "
                          "to be seen only once per iteration if they data structure is not mutated "
                          "during iteration. If key is nil, next returns the first key. If next "
@@ -1027,7 +1066,8 @@ JanetTable *janet_core_env(JanetTable *replacements) {
                          "Propagate a signal from a fiber to the current fiber. The resulting "
                          "stack trace from the current fiber will include frames from fiber. If "
                          "fiber is in a state that can be resumed, resuming the current fiber will "
-                         "first resume fiber."));
+                         "first resume fiber. This function can be used to re-raise an error without "
+                         "losing the original stack trace."));
     janet_quick_asm(env, JANET_FUN_DEBUG,
                     "debug", 1, 0, 1, 1, debug_asm, sizeof(debug_asm),
                     JDOC("(debug &opt x)\n\n"
@@ -1099,7 +1139,7 @@ JanetTable *janet_core_env(JanetTable *replacements) {
                      JDOC("(/ & xs)\n\n"
                           "Returns the quotient of xs. If xs is empty, returns 1. If xs has one value x, returns "
                           "the reciprocal of x. Otherwise return the first value of xs repeatedly divided by the remaining "
-                          "values. Division by two integers uses truncating division."));
+                          "values."));
     templatize_varop(env, JANET_FUN_BAND, "band", -1, -1, JOP_BAND,
                      JDOC("(band & xs)\n\n"
                           "Returns the bit-wise and of all values in xs. Each x in xs must be an integer."));

src/core/emit.c

@@ -37,7 +37,7 @@ int32_t janetc_allocfar(JanetCompiler *c) {
     return reg;
 }
 
-/* Get a register less than 256 */
+/* Get a register less than 256 for temporary use. */
 int32_t janetc_allocnear(JanetCompiler *c, JanetcRegisterTemp tag) {
     return janetc_regalloc_temp(&c->scope->ra, tag);
 }
@@ -205,7 +205,7 @@ static int32_t janetc_regnear(JanetCompiler *c, JanetSlot s, JanetcRegisterTemp
 }
 
 /* Check if two slots are equal */
-static int janetc_sequal(JanetSlot lhs, JanetSlot rhs) {
+int janetc_sequal(JanetSlot lhs, JanetSlot rhs) {
     if ((lhs.flags & ~JANET_SLOTTYPE_ANY) == (rhs.flags & ~JANET_SLOTTYPE_ANY) &&
             lhs.index == rhs.index &&
             lhs.envindex == rhs.envindex) {
@@ -245,8 +245,8 @@ void janetc_copy(
     janetc_moveback(c, dest, nearreg);
     /* Cleanup */
     janetc_regalloc_freetemp(&c->scope->ra, nearreg, JANETC_REGTEMP_3);
-
 }
+
 /* Instruction templated emitters */
 
 static int32_t emit1s(JanetCompiler *c, uint8_t op, JanetSlot s, int32_t rest, int wr) {

src/core/emit.h

@@ -42,6 +42,9 @@ int32_t janetc_emit_ssi(JanetCompiler *c, uint8_t op, JanetSlot s1, JanetSlot s2
 int32_t janetc_emit_ssu(JanetCompiler *c, uint8_t op, JanetSlot s1, JanetSlot s2, uint8_t immediate, int wr);
 int32_t janetc_emit_sss(JanetCompiler *c, uint8_t op, JanetSlot s1, JanetSlot s2, JanetSlot s3, int wr);
 
+/* Check if two slots are equivalent */
+int janetc_sequal(JanetSlot x, JanetSlot y);
+
 /* Move value from one slot to another. Cannot copy to constant slots. */
 void janetc_copy(JanetCompiler *c, JanetSlot dest, JanetSlot src);
 

src/core/features.h

@@ -26,11 +26,15 @@
 #define JANET_FEATURES_H_defined
 
 #ifndef _POSIX_C_SOURCE
-#define _POSIX_C_SOURCE 200112L
+#define _POSIX_C_SOURCE 200809L
+#endif
+
+#if defined(WIN32) || defined(_WIN32)
+#define WIN32_LEAN_AND_MEAN
 #endif
 
 /* Needed for realpath on linux */
-#if !defined(_XOPEN_SOURCE) && defined(__linux__)
+#if !defined(_XOPEN_SOURCE) && (defined(__linux__) || defined(__EMSCRIPTEN__))
 #define _XOPEN_SOURCE 500
 #endif
 

src/core/fiber.c

@@ -218,6 +218,7 @@ int janet_fiber_funcframe(JanetFiber *fiber, JanetFunction *func) {
 static void janet_env_detach(JanetFuncEnv *env) {
     /* Check for closure environment */
     if (env) {
+        janet_env_valid(env);
         int32_t len = env->length;
         size_t s = sizeof(Janet) * (size_t) len;
         Janet *vmem = malloc(s);
@@ -244,10 +245,38 @@ static void janet_env_detach(JanetFuncEnv *env) {
     }
 }
 
+/* Validate potentially untrusted func env (unmarshalled envs are difficult to verify) */
+int janet_env_valid(JanetFuncEnv *env) {
+    if (env->offset < 0) {
+        int32_t real_offset = -(env->offset);
+        JanetFiber *fiber = env->as.fiber;
+        int32_t i = fiber->frame;
+        while (i > 0) {
+            JanetStackFrame *frame = (JanetStackFrame *)(fiber->data + i - JANET_FRAME_SIZE);
+            if (real_offset == i &&
+                    frame->env == env &&
+                    frame->func &&
+                    frame->func->def->slotcount == env->length) {
+                env->offset = real_offset;
+                return 1;
+            }
+            i = frame->prevframe;
+        }
+        /* Invalid, set to empty off-stack variant. */
+        env->offset = 0;
+        env->length = 0;
+        env->as.values = NULL;
+        return 0;
+    } else {
+        return 1;
+    }
+}
+
 /* Detach a fiber from the env if the target fiber has stopped mutating */
 void janet_env_maybe_detach(JanetFuncEnv *env) {
     /* Check for detachable closure envs */
-    if (env->offset) {
+    janet_env_valid(env);
+    if (env->offset > 0) {
         JanetFiberStatus s = janet_fiber_status(env->as.fiber);
         int isFinished = s == JANET_STATUS_DEAD ||
                          s == JANET_STATUS_ERROR ||
@@ -376,6 +405,10 @@ JanetFiber *janet_current_fiber(void) {
     return janet_vm_fiber;
 }
 
+JanetFiber *janet_root_fiber(void) {
+    return janet_vm_root_fiber;
+}
+
 /* CFuns */
 
 static Janet cfun_fiber_getenv(int32_t argc, Janet *argv) {
@@ -416,7 +449,7 @@ static Janet cfun_fiber_new(int32_t argc, Janet *argv) {
             } else {
                 switch (view.bytes[i]) {
                     default:
-                        janet_panicf("invalid flag %c, expected a, d, e, u, or y", view.bytes[i]);
+                        janet_panicf("invalid flag %c, expected a, t, d, e, u, y, i, or p", view.bytes[i]);
                         break;
                     case 'a':
                         fiber->flags |=
@@ -479,6 +512,12 @@ static Janet cfun_fiber_current(int32_t argc, Janet *argv) {
     return janet_wrap_fiber(janet_vm_fiber);
 }
 
+static Janet cfun_fiber_root(int32_t argc, Janet *argv) {
+    (void) argv;
+    janet_fixarity(argc, 0);
+    return janet_wrap_fiber(janet_vm_root_fiber);
+}
+
 static Janet cfun_fiber_maxstack(int32_t argc, Janet *argv) {
     janet_fixarity(argc, 1);
     JanetFiber *fiber = janet_getfiber(argv, 0);
@@ -546,6 +585,12 @@ static const JanetReg fiber_cfuns[] = {
              "\t:alive - the fiber is currently running and cannot be resumed\n"
              "\t:new - the fiber has just been created and not yet run")
     },
+    {
+        "fiber/root", cfun_fiber_root,
+        JDOC("(fiber/root)\n\n"
+             "Returns the current root fiber. The root fiber is the oldest ancestor "
+             "that does not have a parent.")
+    },
     {
         "fiber/current", cfun_fiber_current,
         JDOC("(fiber/current)\n\n"

src/core/fiber.h

@@ -74,5 +74,6 @@ int janet_fiber_funcframe_tail(JanetFiber *fiber, JanetFunction *func);
 void janet_fiber_cframe(JanetFiber *fiber, JanetCFunction cfun);
 void janet_fiber_popframe(JanetFiber *fiber);
 void janet_env_maybe_detach(JanetFuncEnv *env);
+int janet_env_valid(JanetFuncEnv *env);
 
 #endif

src/core/gc.c

@@ -39,6 +39,7 @@ struct JanetScratch {
 JANET_THREAD_LOCAL void *janet_vm_blocks;
 JANET_THREAD_LOCAL size_t janet_vm_gc_interval;
 JANET_THREAD_LOCAL size_t janet_vm_next_collection;
+JANET_THREAD_LOCAL size_t janet_vm_block_count;
 JANET_THREAD_LOCAL int janet_vm_gc_suspend = 0;
 
 /* Roots */
@@ -193,7 +194,7 @@ static void janet_mark_funcenv(JanetFuncEnv *env) {
     /* If closure env references a dead fiber, we can just copy out the stack frame we need so
      * we don't need to keep around the whole dead fiber. */
     janet_env_maybe_detach(env);
-    if (env->offset) {
+    if (env->offset > 0) {
         /* On stack */
         janet_mark_fiber(env->as.fiber);
     } else {
@@ -327,6 +328,7 @@ void janet_sweep() {
             previous = current;
             current->flags &= ~JANET_MEM_REACHABLE;
         } else {
+            janet_vm_block_count--;
             janet_deinit_block(current);
             if (NULL != previous) {
                 previous->next = next;
@@ -359,6 +361,7 @@ void *janet_gcalloc(enum JanetMemoryType type, size_t size) {
     janet_vm_next_collection += size;
     mem->next = janet_vm_blocks;
     janet_vm_blocks = mem;
+    janet_vm_block_count++;
 
     return (void *)mem;
 }
@@ -388,7 +391,18 @@ void janet_collect(void) {
     uint32_t i;
     if (janet_vm_gc_suspend) return;
     depth = JANET_RECURSION_GUARD;
+    /* Try and prevent many major collections back to back.
+     * A full collection will take O(janet_vm_block_count) time.
+     * If we have a large heap, make sure our interval is not too
+     * small so we won't make many collections over it. This is just a
+     * heuristic for automatically changing the gc interval */
+    if (janet_vm_block_count * 8 > janet_vm_gc_interval) {
+        janet_vm_gc_interval = janet_vm_block_count * sizeof(JanetGCObject);
+    }
     orig_rootcount = janet_vm_root_count;
+#ifdef JANET_NET
+    janet_net_markloop();
+#endif
     for (i = 0; i < orig_rootcount; i++)
         janet_mark(janet_vm_roots[i]);
     while (orig_rootcount < janet_vm_root_count) {
@@ -530,7 +544,7 @@ void *janet_srealloc(void *mem, size_t size) {
             if (i == 0) break;
         }
     }
-    janet_exit("invalid janet_srealloc");
+    JANET_EXIT("invalid janet_srealloc");
 }
 
 void janet_sfinalizer(void *mem, JanetScratchFinalizer finalizer) {
@@ -551,5 +565,5 @@ void janet_sfree(void *mem) {
             if (i == 0) break;
         }
     }
-    janet_exit("invalid janet_sfree");
+    JANET_EXIT("invalid janet_sfree");
 }

src/core/inttypes.c

@@ -20,18 +20,18 @@
 * IN THE SOFTWARE.
 */
 
-#include <errno.h>
-#include <stdlib.h>
-#include <limits.h>
-#include <inttypes.h>
-#include <math.h>
-
 #ifndef JANET_AMALG
 #include "features.h"
 #include <janet.h>
 #include "util.h"
 #endif
 
+#include <errno.h>
+#include <stdlib.h>
+#include <limits.h>
+#include <inttypes.h>
+#include <math.h>
+
 /* Conditional compilation */
 #ifdef JANET_INT_TYPES
 
@@ -197,6 +197,122 @@ static Janet cfun_it_u64_new(int32_t argc, Janet *argv) {
     return janet_wrap_u64(janet_unwrap_u64(argv[0]));
 }
 
+// Code to support polymorphic comparison.
+//
+// int/u64 and int/s64 support a "compare" method that allows
+// comparison to each other, and to Janet numbers, using the
+// "compare" "compare<" ... functions.
+//
+// In the following code explicit casts are sometimes used to help
+// make it clear when int/float conversions are happening.
+//
+static int compare_double_double(double x, double y) {
+    return (x < y) ? -1 : ((x > y) ? 1 : 0);
+}
+
+static int compare_int64_double(int64_t x, double y) {
+    if (isnan(y)) {
+        return 0; // clojure and python do this
+    } else if ((y > (- ((double) MAX_INT_IN_DBL))) && (y < ((double) MAX_INT_IN_DBL))) {
+        double dx = (double) x;
+        return compare_double_double(dx, y);
+    } else if (y > ((double) INT64_MAX)) {
+        return -1;
+    } else if (y < ((double) INT64_MIN)) {
+        return 1;
+    } else {
+        int64_t yi = (int64_t) y;
+        return (x < yi) ? -1 : ((x > yi) ? 1 : 0);
+    }
+}
+
+static int compare_uint64_double(uint64_t x, double y) {
+    if (isnan(y)) {
+        return 0; // clojure and python do this
+    } else if (y < 0) {
+        return 1;
+    } else if ((y >= 0) && (y < ((double) MAX_INT_IN_DBL))) {
+        double dx = (double) x;
+        return compare_double_double(dx, y);
+    } else if (y > ((double) UINT64_MAX)) {
+        return -1;
+    } else {
+        uint64_t yi = (uint64_t) y;
+        return (x < yi) ? -1 : ((x > yi) ? 1 : 0);
+    }
+}
+
+
+static Janet cfun_it_s64_compare(int32_t argc, Janet *argv) {
+    janet_fixarity(argc, 2);
+    if (janet_is_int(argv[0]) != JANET_INT_S64)
+        janet_panic("compare method requires int/s64 as first argument");
+    int64_t x = janet_unwrap_s64(argv[0]);
+    switch (janet_type(argv[1])) {
+        default:
+            break;
+        case JANET_NUMBER : {
+            double y = janet_unwrap_number(argv[1]);
+            return janet_wrap_number(compare_int64_double(x, y));
+        }
+        case JANET_ABSTRACT: {
+            void *abst = janet_unwrap_abstract(argv[1]);
+            if (janet_abstract_type(abst) == &janet_s64_type) {
+                int64_t y = *(int64_t *)abst;
+                return janet_wrap_number((x < y) ? -1 : (x > y ? 1 : 0));
+            } else if (janet_abstract_type(abst) == &janet_u64_type) {
+                // comparing signed to unsigned -- be careful!
+                uint64_t y = *(uint64_t *)abst;
+                if (x < 0) {
+                    return janet_wrap_number(-1);
+                } else if (y > INT64_MAX) {
+                    return janet_wrap_number(-1);
+                } else {
+                    int64_t y2 = (int64_t) y;
+                    return janet_wrap_number((x < y2) ? -1 : (x > y2 ? 1 : 0));
+                }
+            }
+            break;
+        }
+    }
+    return janet_wrap_nil();
+}
+
+static Janet cfun_it_u64_compare(int32_t argc, Janet *argv) {
+    janet_fixarity(argc, 2);
+    if (janet_is_int(argv[0]) != JANET_INT_U64)  // is this needed?
+        janet_panic("compare method requires int/u64 as first argument");
+    uint64_t x = janet_unwrap_u64(argv[0]);
+    switch (janet_type(argv[1])) {
+        default:
+            break;
+        case JANET_NUMBER : {
+            double y = janet_unwrap_number(argv[1]);
+            return janet_wrap_number(compare_uint64_double(x, y));
+        }
+        case JANET_ABSTRACT: {
+            void *abst = janet_unwrap_abstract(argv[1]);
+            if (janet_abstract_type(abst) == &janet_u64_type) {
+                uint64_t y = *(uint64_t *)abst;
+                return janet_wrap_number((x < y) ? -1 : (x > y ? 1 : 0));
+            } else if (janet_abstract_type(abst) == &janet_s64_type) {
+                // comparing unsigned to signed -- be careful!
+                int64_t y = *(int64_t *)abst;
+                if (y < 0) {
+                    return janet_wrap_number(1);
+                } else if (x > INT64_MAX) {
+                    return janet_wrap_number(1);
+                } else {
+                    int64_t x2 = (int64_t) x;
+                    return janet_wrap_number((x2 < y) ? -1 : (x2 > y ? 1 : 0));
+                }
+            }
+            break;
+        }
+    }
+    return janet_wrap_nil();
+}
+
 #define OPMETHOD(T, type, name, oper) \
 static Janet cfun_it_##type##_##name(int32_t argc, Janet *argv) { \
     janet_arity(argc, 2, -1); \
@@ -266,14 +382,6 @@ static Janet cfun_it_##type##_##name(int32_t argc, Janet *argv) { \
     return janet_wrap_abstract(box); \
 } \
 
-#define COMPMETHOD(T, type, name, oper) \
-static Janet cfun_it_##type##_##name(int32_t argc, Janet *argv) { \
-    janet_fixarity(argc, 2); \
-    T v1 = janet_unwrap_##type(argv[0]); \
-    T v2 = janet_unwrap_##type(argv[1]); \
-    return janet_wrap_boolean(v1 oper v2); \
-}
-
 static Janet cfun_it_s64_mod(int32_t argc, Janet *argv) {
     janet_arity(argc, 2, -1);
     int64_t *box = janet_abstract(&janet_s64_type, sizeof(int64_t));
@@ -316,13 +424,6 @@ OPMETHOD(int64_t, s64, or, |)
 OPMETHOD(int64_t, s64, xor, ^)
 OPMETHOD(int64_t, s64, lshift, <<)
 OPMETHOD(int64_t, s64, rshift, >>)
-COMPMETHOD(int64_t, s64, lt, <)
-COMPMETHOD(int64_t, s64, gt, >)
-COMPMETHOD(int64_t, s64, le, <=)
-COMPMETHOD(int64_t, s64, ge, >=)
-COMPMETHOD(int64_t, s64, eq, ==)
-COMPMETHOD(int64_t, s64, ne, !=)
-
 OPMETHOD(uint64_t, u64, add, +)
 OPMETHOD(uint64_t, u64, sub, -)
 OPMETHODINVERT(uint64_t, u64, subi, -)
@@ -336,18 +437,13 @@ OPMETHOD(uint64_t, u64, or, |)
 OPMETHOD(uint64_t, u64, xor, ^)
 OPMETHOD(uint64_t, u64, lshift, <<)
 OPMETHOD(uint64_t, u64, rshift, >>)
-COMPMETHOD(uint64_t, u64, lt, <)
-COMPMETHOD(uint64_t, u64, gt, >)
-COMPMETHOD(uint64_t, u64, le, <=)
-COMPMETHOD(uint64_t, u64, ge, >=)
-COMPMETHOD(uint64_t, u64, eq, ==)
-COMPMETHOD(uint64_t, u64, ne, !=)
 
 #undef OPMETHOD
 #undef DIVMETHOD
 #undef DIVMETHOD_SIGNED
 #undef COMPMETHOD
 
+
 static JanetMethod it_s64_methods[] = {
     {"+", cfun_it_s64_add},
     {"r+", cfun_it_s64_add},
@@ -361,12 +457,6 @@ static JanetMethod it_s64_methods[] = {
     {"rmod", cfun_it_s64_modi},
     {"%", cfun_it_s64_rem},
     {"r%", cfun_it_s64_remi},
-    {"<", cfun_it_s64_lt},
-    {">", cfun_it_s64_gt},
-    {"<=", cfun_it_s64_le},
-    {">=", cfun_it_s64_ge},
-    {"=", cfun_it_s64_eq},
-    {"!=", cfun_it_s64_ne},
     {"&", cfun_it_s64_and},
     {"r&", cfun_it_s64_and},
     {"|", cfun_it_s64_or},
@@ -375,6 +465,7 @@ static JanetMethod it_s64_methods[] = {
     {"r^", cfun_it_s64_xor},
     {"<<", cfun_it_s64_lshift},
     {">>", cfun_it_s64_rshift},
+    {"compare", cfun_it_s64_compare},
 
     {NULL, NULL}
 };
@@ -392,12 +483,6 @@ static JanetMethod it_u64_methods[] = {
     {"rmod", cfun_it_u64_modi},
     {"%", cfun_it_u64_mod},
     {"r%", cfun_it_u64_modi},
-    {"<", cfun_it_u64_lt},
-    {">", cfun_it_u64_gt},
-    {"<=", cfun_it_u64_le},
-    {">=", cfun_it_u64_ge},
-    {"=", cfun_it_u64_eq},
-    {"!=", cfun_it_u64_ne},
     {"&", cfun_it_u64_and},
     {"r&", cfun_it_u64_and},
     {"|", cfun_it_u64_or},
@@ -406,6 +491,7 @@ static JanetMethod it_u64_methods[] = {
     {"r^", cfun_it_u64_xor},
     {"<<", cfun_it_u64_lshift},
     {">>", cfun_it_u64_rshift},
+    {"compare", cfun_it_u64_compare},
 
     {NULL, NULL}
 };

src/core/io.c

@@ -30,23 +30,30 @@
 #include <errno.h>
 
 #ifndef JANET_WINDOWS
+#include <fcntl.h>
 #include <sys/wait.h>
+#include <unistd.h>
 #endif
 
 static int cfun_io_gc(void *p, size_t len);
 static int io_file_get(void *p, Janet key, Janet *out);
+static void io_file_marshal(void *p, JanetMarshalContext *ctx);
+static void *io_file_unmarshal(JanetMarshalContext *ctx);
 
 const JanetAbstractType janet_file_type = {
     "core/file",
     cfun_io_gc,
     NULL,
     io_file_get,
-    JANET_ATEND_GET
+    NULL,
+    io_file_marshal,
+    io_file_unmarshal,
+    JANET_ATEND_UNMARSHAL
 };
 
 /* Check arguments to fopen */
-static int checkflags(const uint8_t *str) {
-    int flags = 0;
+static int32_t checkflags(const uint8_t *str) {
+    int32_t flags = 0;
     int32_t i;
     int32_t len = janet_string_length(str);
     if (!len || len > 3)
@@ -83,27 +90,26 @@ static int checkflags(const uint8_t *str) {
     return flags;
 }
 
-static Janet makef(FILE *f, int flags) {
+static void *makef(FILE *f, int32_t flags) {
     JanetFile *iof = (JanetFile *) janet_abstract(&janet_file_type, sizeof(JanetFile));
     iof->file = f;
     iof->flags = flags;
-    return janet_wrap_abstract(iof);
+#ifndef JANET_WINDOWS
+    /* While we would like fopen to set cloexec by default (like O_CLOEXEC) with the e flag, that is
+     * not standard. */
+    if (!(flags & JANET_FILE_NOT_CLOSEABLE))
+        fcntl(fileno(f), F_SETFD, FD_CLOEXEC);
+#endif
+    return iof;
 }
 
 /* Open a process */
-#ifdef __EMSCRIPTEN__
-static Janet cfun_io_popen(int32_t argc, Janet *argv) {
-    (void) argc;
-    (void) argv;
-    janet_panic("not implemented on this platform");
-    return janet_wrap_nil();
-}
-#else
+#ifndef JANET_NO_PROCESSES
 static Janet cfun_io_popen(int32_t argc, Janet *argv) {
     janet_arity(argc, 1, 2);
     const uint8_t *fname = janet_getstring(argv, 0);
     const uint8_t *fmode = NULL;
-    int flags;
+    int32_t flags;
     if (argc == 2) {
         fmode = janet_getkeyword(argv, 1);
         if (janet_string_length(fmode) != 1 ||
@@ -122,13 +128,14 @@ static Janet cfun_io_popen(int32_t argc, Janet *argv) {
     if (!f) {
         return janet_wrap_nil();
     }
-    return makef(f, flags);
+    return janet_makefile(f, flags);
 }
 #endif
 
 static Janet cfun_io_temp(int32_t argc, Janet *argv) {
     (void)argv;
     janet_fixarity(argc, 0);
+    // XXX use mkostemp when we can to avoid CLOEXEC race.
     FILE *tmp = tmpfile();
     if (!tmp)
         janet_panicf("unable to create temporary file - %s", strerror(errno));
@@ -139,7 +146,7 @@ static Janet cfun_io_fopen(int32_t argc, Janet *argv) {
     janet_arity(argc, 1, 2);
     const uint8_t *fname = janet_getstring(argv, 0);
     const uint8_t *fmode;
-    int flags;
+    int32_t flags;
     if (argc == 2) {
         fmode = janet_getkeyword(argv, 1);
         flags = checkflags(fmode);
@@ -148,7 +155,7 @@ static Janet cfun_io_fopen(int32_t argc, Janet *argv) {
         flags = JANET_FILE_READ;
     }
     FILE *f = fopen((const char *)fname, (const char *)fmode);
-    return f ? makef(f, flags) : janet_wrap_nil();
+    return f ? janet_makefile(f, flags) : janet_wrap_nil();
 }
 
 /* Read up to n bytes into buffer. */
@@ -239,12 +246,24 @@ static Janet cfun_io_fflush(int32_t argc, Janet *argv) {
     return argv[0];
 }
 
+#ifdef JANET_WINDOWS
+#define pclose _pclose
+#define WEXITSTATUS(x) x
+#endif
+
 /* Cleanup a file */
 static int cfun_io_gc(void *p, size_t len) {
     (void) len;
     JanetFile *iof = (JanetFile *)p;
     if (!(iof->flags & (JANET_FILE_NOT_CLOSEABLE | JANET_FILE_CLOSED))) {
-        return fclose(iof->file);
+        /* We can't panic inside a gc, so just ignore bad statuses here */
+        if (iof->flags & JANET_FILE_PIPED) {
+#ifndef JANET_NO_PROCESSES
+            pclose(iof->file);
+#endif
+        } else {
+            fclose(iof->file);
+        }
     }
     return 0;
 }
@@ -258,16 +277,17 @@ static Janet cfun_io_fclose(int32_t argc, Janet *argv) {
     if (iof->flags & (JANET_FILE_NOT_CLOSEABLE))
         janet_panic("file not closable");
     if (iof->flags & JANET_FILE_PIPED) {
-#ifdef JANET_WINDOWS
-#define pclose _pclose
-#define WEXITSTATUS(x) x
-#endif
+#ifndef JANET_NO_PROCESSES
         int status = pclose(iof->file);
         iof->flags |= JANET_FILE_CLOSED;
         if (status == -1) janet_panic("could not close file");
         return janet_wrap_integer(WEXITSTATUS(status));
+#endif
     } else {
-        if (fclose(iof->file)) janet_panic("could not close file");
+        if (fclose(iof->file)) {
+            iof->flags |= JANET_FILE_NOT_CLOSEABLE;
+            janet_panic("could not close file");
+        }
         iof->flags |= JANET_FILE_CLOSED;
         return janet_wrap_nil();
     }
@@ -316,6 +336,50 @@ static int io_file_get(void *p, Janet key, Janet *out) {
     return janet_getmethod(janet_unwrap_keyword(key), io_file_methods, out);
 }
 
+static void io_file_marshal(void *p, JanetMarshalContext *ctx) {
+    JanetFile *iof = (JanetFile *)p;
+    if (ctx->flags & JANET_MARSHAL_UNSAFE) {
+        janet_marshal_abstract(ctx, p);
+#ifdef JANET_WINDOWS
+        janet_marshal_int(ctx, _fileno(iof->file));
+#else
+        janet_marshal_int(ctx, fileno(iof->file));
+#endif
+        janet_marshal_int(ctx, iof->flags);
+    } else {
+        janet_panic("cannot marshal file in safe mode");
+    }
+}
+
+static void *io_file_unmarshal(JanetMarshalContext *ctx) {
+    if (ctx->flags & JANET_MARSHAL_UNSAFE) {
+        JanetFile *iof = janet_unmarshal_abstract(ctx, sizeof(JanetFile));
+        int32_t fd = janet_unmarshal_int(ctx);
+        int32_t flags = janet_unmarshal_int(ctx);
+        char fmt[4] = {0};
+        int index = 0;
+        if (flags & JANET_FILE_READ) fmt[index++] = 'r';
+        if (flags & JANET_FILE_APPEND) {
+            fmt[index++] = 'a';
+        } else if (flags & JANET_FILE_WRITE) {
+            fmt[index++] = 'w';
+        }
+#ifdef JANET_WINDOWS
+        iof->file = _fdopen(fd, fmt);
+#else
+        iof->file = fdopen(fd, fmt);
+#endif
+        if (iof->file == NULL) {
+            iof->flags = JANET_FILE_CLOSED;
+        } else {
+            iof->flags = flags;
+        }
+        return iof;
+    } else {
+        janet_panic("cannot unmarshal file in safe mode");
+    }
+}
+
 FILE *janet_dynfile(const char *name, FILE *def) {
     Janet x = janet_dyn(name);
     if (!janet_checktype(x, JANET_ABSTRACT)) return def;
@@ -502,7 +566,7 @@ void janet_dynprintf(const char *name, FILE *dflt_file, const char *format, ...)
             int32_t len = 0;
             while (format[len]) len++;
             janet_buffer_init(&buffer, len);
-            janet_formatb(&buffer, format, args);
+            janet_formatbv(&buffer, format, args);
             if (xtype == JANET_ABSTRACT) {
                 void *abstract = janet_unwrap_abstract(x);
                 if (janet_abstract_type(abstract) != &janet_file_type)
@@ -515,7 +579,7 @@ void janet_dynprintf(const char *name, FILE *dflt_file, const char *format, ...)
             break;
         }
         case JANET_BUFFER:
-            janet_formatb(janet_unwrap_buffer(x), format, args);
+            janet_formatbv(janet_unwrap_buffer(x), format, args);
             break;
     }
     va_end(args);
@@ -640,6 +704,7 @@ static const JanetReg io_cfuns[] = {
              "for the relative number of bytes to seek in the file. n may be a real "
              "number to handle large files of more the 4GB. Returns the file handle.")
     },
+#ifndef JANET_NO_PROCESSES
     {
         "file/popen", cfun_io_popen,
         JDOC("(file/popen path &opt mode)\n\n"
@@ -648,6 +713,7 @@ static const JanetReg io_cfuns[] = {
              "process can be read from the file. In :w mode, the stdin of the process "
              "can be written to. Returns the new file.")
     },
+#endif
     {NULL, NULL, NULL}
 };
 
@@ -660,7 +726,7 @@ FILE *janet_getfile(const Janet *argv, int32_t n, int *flags) {
 }
 
 Janet janet_makefile(FILE *f, int flags) {
-    return makef(f, flags);
+    return janet_wrap_abstract(makef(f, flags));
 }
 
 JanetAbstract janet_checkfile(Janet j) {
@@ -676,18 +742,18 @@ FILE *janet_unwrapfile(Janet j, int *flags) {
 /* Module entry point */
 void janet_lib_io(JanetTable *env) {
     janet_core_cfuns(env, NULL, io_cfuns);
-
+    janet_register_abstract_type(&janet_file_type);
     /* stdout */
     janet_core_def(env, "stdout",
-                   makef(stdout, JANET_FILE_APPEND | JANET_FILE_NOT_CLOSEABLE | JANET_FILE_SERIALIZABLE),
+                   janet_makefile(stdout, JANET_FILE_APPEND | JANET_FILE_NOT_CLOSEABLE | JANET_FILE_SERIALIZABLE),
                    JDOC("The standard output file."));
     /* stderr */
     janet_core_def(env, "stderr",
-                   makef(stderr, JANET_FILE_APPEND | JANET_FILE_NOT_CLOSEABLE | JANET_FILE_SERIALIZABLE),
+                   janet_makefile(stderr, JANET_FILE_APPEND | JANET_FILE_NOT_CLOSEABLE | JANET_FILE_SERIALIZABLE),
                    JDOC("The standard error file."));
     /* stdin */
     janet_core_def(env, "stdin",
-                   makef(stdin, JANET_FILE_READ | JANET_FILE_NOT_CLOSEABLE | JANET_FILE_SERIALIZABLE),
+                   janet_makefile(stdin, JANET_FILE_READ | JANET_FILE_NOT_CLOSEABLE | JANET_FILE_SERIALIZABLE),
                    JDOC("The standard input file."));
 
 }

src/core/marsh.c

@@ -42,26 +42,28 @@ typedef struct {
 /* Lead bytes in marshaling protocol */
 enum {
     LB_REAL = 200,
-    LB_NIL,
-    LB_FALSE,
-    LB_TRUE,
-    LB_FIBER,
-    LB_INTEGER,
-    LB_STRING,
-    LB_SYMBOL,
-    LB_KEYWORD,
-    LB_ARRAY,
-    LB_TUPLE,
-    LB_TABLE,
-    LB_TABLE_PROTO,
-    LB_STRUCT,
-    LB_BUFFER,
-    LB_FUNCTION,
-    LB_REGISTRY,
-    LB_ABSTRACT,
-    LB_REFERENCE,
-    LB_FUNCENV_REF,
-    LB_FUNCDEF_REF
+    LB_NIL, /* 201 */
+    LB_FALSE, /* 202 */
+    LB_TRUE,  /* 203 */
+    LB_FIBER, /* 204 */
+    LB_INTEGER, /* 205 */
+    LB_STRING, /* 206 */
+    LB_SYMBOL, /* 207 */
+    LB_KEYWORD, /* 208 */
+    LB_ARRAY, /* 209 */
+    LB_TUPLE, /* 210 */
+    LB_TABLE, /* 211 */
+    LB_TABLE_PROTO, /* 212 */
+    LB_STRUCT, /* 213 */
+    LB_BUFFER, /* 214 */
+    LB_FUNCTION, /* 215 */
+    LB_REGISTRY, /* 216 */
+    LB_ABSTRACT, /* 217 */
+    LB_REFERENCE, /* 218 */
+    LB_FUNCENV_REF, /* 219 */
+    LB_FUNCDEF_REF, /* 220 */
+    LB_UNSAFE_CFUNCTION, /* 221 */
+    LB_UNSAFE_POINTER /* 222 */
 } LeadBytes;
 
 /* Helper to look inside an entry in an environment */
@@ -183,8 +185,9 @@ static void marshal_one_env(MarshalState *st, JanetFuncEnv *env, int flags) {
             return;
         }
     }
+    janet_env_valid(env);
     janet_v_push(st->seen_envs, env);
-    if (env->offset && (JANET_STATUS_ALIVE == janet_fiber_status(env->as.fiber))) {
+    if (env->offset > 0 && (JANET_STATUS_ALIVE == janet_fiber_status(env->as.fiber))) {
         pushint(st, 0);
         pushint(st, env->length);
         Janet *values = env->as.fiber->data + env->offset;
@@ -200,7 +203,7 @@ static void marshal_one_env(MarshalState *st, JanetFuncEnv *env, int flags) {
         janet_env_maybe_detach(env);
         pushint(st, env->offset);
         pushint(st, env->length);
-        if (env->offset) {
+        if (env->offset > 0) {
             /* On stack variant */
             marshal_one(st, janet_wrap_fiber(env->as.fiber), flags + 1);
         } else {
@@ -211,15 +214,6 @@ static void marshal_one_env(MarshalState *st, JanetFuncEnv *env, int flags) {
     }
 }
 
-/* Add function flags to janet functions */
-static void janet_func_addflags(JanetFuncDef *def) {
-    if (def->name) def->flags |= JANET_FUNCDEF_FLAG_HASNAME;
-    if (def->source) def->flags |= JANET_FUNCDEF_FLAG_HASSOURCE;
-    if (def->defs) def->flags |= JANET_FUNCDEF_FLAG_HASDEFS;
-    if (def->environments) def->flags |= JANET_FUNCDEF_FLAG_HASENVS;
-    if (def->sourcemap) def->flags |= JANET_FUNCDEF_FLAG_HASSOURCEMAP;
-}
-
 /* Marshal a sequence of u32s */
 static void janet_marshal_u32s(MarshalState *st, const uint32_t *u32s, int32_t n) {
     for (int32_t i = 0; i < n; i++) {
@@ -240,7 +234,6 @@ static void marshal_one_def(MarshalState *st, JanetFuncDef *def, int flags) {
             return;
         }
     }
-    janet_func_addflags(def);
     /* Add to lookup */
     janet_v_push(st->seen_defs, def);
     pushint(st, def->flags);
@@ -562,9 +555,25 @@ static void marshal_one(MarshalState *st, Janet x, int flags) {
             marshal_one_fiber(st, janet_unwrap_fiber(x), flags + 1);
             return;
         }
+        case JANET_CFUNCTION: {
+            if (!(flags & JANET_MARSHAL_UNSAFE)) goto no_registry;
+            MARK_SEEN();
+            pushbyte(st, LB_UNSAFE_CFUNCTION);
+            JanetCFunction cfn = janet_unwrap_cfunction(x);
+            pushbytes(st, (uint8_t *) &cfn, sizeof(JanetCFunction));
+            return;
+        }
+        case JANET_POINTER: {
+            if (!(flags & JANET_MARSHAL_UNSAFE)) goto no_registry;
+            MARK_SEEN();
+            pushbyte(st, LB_UNSAFE_POINTER);
+            void *ptr = janet_unwrap_pointer(x);
+            pushbytes(st, (uint8_t *) &ptr, sizeof(void *));
+            return;
+        }
+    no_registry:
         default: {
             janet_panicf("no registry value and cannot marshal %p", x);
-            return;
         }
     }
 #undef MARK_SEEN
@@ -634,6 +643,15 @@ static int32_t readint(UnmarshalState *st, const uint8_t **atdata) {
     return ret;
 }
 
+/* Helper to read a natural number (int >= 0). */
+static int32_t readnat(UnmarshalState *st, const uint8_t **atdata) {
+    int32_t ret = readint(st, atdata);
+    if (ret < 0) {
+        janet_panicf("expected integer >= 0, got %d", ret);
+    }
+    return ret;
+}
+
 /* Helper to read a size_t (up to 8 bytes unsigned). */
 static uint64_t read64(UnmarshalState *st, const uint8_t **atdata) {
     uint64_t ret;
@@ -702,30 +720,31 @@ static const uint8_t *unmarshal_one_env(
         JanetFuncEnv *env = janet_gcalloc(JANET_MEMORY_FUNCENV, sizeof(JanetFuncEnv));
         env->length = 0;
         env->offset = 0;
+        env->as.values = NULL;
         janet_v_push(st->lookup_envs, env);
-        int32_t offset = readint(st, &data);
-        int32_t length = readint(st, &data);
-        if (offset) {
+        int32_t offset = readnat(st, &data);
+        int32_t length = readnat(st, &data);
+        if (offset > 0) {
             Janet fiberv;
             /* On stack variant */
             data = unmarshal_one(st, data, &fiberv, flags);
             janet_asserttype(fiberv, JANET_FIBER);
             env->as.fiber = janet_unwrap_fiber(fiberv);
-            /* Unmarshalling fiber may set values */
-            if (env->offset != 0 && env->offset != offset)
-                janet_panic("invalid funcenv offset");
-            if (env->length != 0 && env->length != length)
-                janet_panic("invalid funcenv length");
+            /* Negative offset indicates untrusted input */
+            env->offset = -offset;
         } else {
             /* Off stack variant */
+            if (length == 0) {
+                janet_panic("invalid funcenv length");
+            }
             env->as.values = malloc(sizeof(Janet) * (size_t) length);
             if (!env->as.values) {
                 JANET_OUT_OF_MEMORY;
             }
+            env->offset = 0;
             for (int32_t i = 0; i < length; i++)
                 data = unmarshal_one(st, data, env->as.values + i, flags);
         }
-        env->offset = offset;
         env->length = length;
         *out = env;
     }
@@ -770,6 +789,11 @@ static const uint8_t *unmarshal_one_def(
         def->name = NULL;
         def->source = NULL;
         def->closure_bitset = NULL;
+        def->defs = NULL;
+        def->environments = NULL;
+        def->constants = NULL;
+        def->bytecode = NULL;
+        def->sourcemap = NULL;
         janet_v_push(st->lookup_defs, def);
 
         /* Set default lengths to zero */
@@ -780,18 +804,18 @@ static const uint8_t *unmarshal_one_def(
 
         /* Read flags and other fixed values */
         def->flags = readint(st, &data);
-        def->slotcount = readint(st, &data);
-        def->arity = readint(st, &data);
-        def->min_arity = readint(st, &data);
-        def->max_arity = readint(st, &data);
+        def->slotcount = readnat(st, &data);
+        def->arity = readnat(st, &data);
+        def->min_arity = readnat(st, &data);
+        def->max_arity = readnat(st, &data);
 
         /* Read some lengths */
-        constants_length = readint(st, &data);
-        bytecode_length = readint(st, &data);
+        constants_length = readnat(st, &data);
+        bytecode_length = readnat(st, &data);
         if (def->flags & JANET_FUNCDEF_FLAG_HASENVS)
-            environments_length = readint(st, &data);
+            environments_length = readnat(st, &data);
         if (def->flags & JANET_FUNCDEF_FLAG_HASDEFS)
-            defs_length = readint(st, &data);
+            defs_length = readnat(st, &data);
 
         /* Check name and source (optional) */
         if (def->flags & JANET_FUNCDEF_FLAG_HASNAME) {
@@ -874,11 +898,12 @@ static const uint8_t *unmarshal_one_def(
 
         /* Unmarshal closure bitset if needed */
         if (def->flags & JANET_FUNCDEF_FLAG_HASCLOBITSET) {
-            def->closure_bitset = malloc(sizeof(uint32_t) * def->slotcount);
+            int32_t n = (def->slotcount + 31) >> 5;
+            def->closure_bitset = malloc(sizeof(uint32_t) * (size_t) n);
             if (NULL == def->closure_bitset) {
                 JANET_OUT_OF_MEMORY;
             }
-            data = janet_unmarshal_u32s(st, data, def->closure_bitset, (def->slotcount + 31) >> 5);
+            data = janet_unmarshal_u32s(st, data, def->closure_bitset, n);
         }
 
         /* Validate */
@@ -898,7 +923,7 @@ static const uint8_t *unmarshal_one_fiber(
     JanetFiber **out,
     int flags) {
 
-    /* Initialize a new fiber */
+    /* Initialize a new fiber with gc friendly defaults */
     JanetFiber *fiber = janet_gcalloc(JANET_MEMORY_FIBER, sizeof(JanetFiber));
     fiber->flags = 0;
     fiber->frame = 0;
@@ -913,42 +938,41 @@ static const uint8_t *unmarshal_one_fiber(
     /* Push fiber to seen stack */
     janet_v_push(st->lookup, janet_wrap_fiber(fiber));
 
-    /* Set frame later so fiber can be GCed at anytime if unmarshalling fails */
-    int32_t frame = 0;
-    int32_t stack = 0;
-    int32_t stacktop = 0;
-
     /* Read ints */
-    fiber->flags = readint(st, &data);
-    frame = readint(st, &data);
-    fiber->stackstart = readint(st, &data);
-    fiber->stacktop = readint(st, &data);
-    fiber->maxstack = readint(st, &data);
+    int32_t fiber_flags = readint(st, &data);
+    int32_t frame = readnat(st, &data);
+    int32_t fiber_stackstart = readnat(st, &data);
+    int32_t fiber_stacktop = readnat(st, &data);
+    int32_t fiber_maxstack = readnat(st, &data);
+    JanetTable *fiber_env = NULL;
 
     /* Check for bad flags and ints */
-    if ((int32_t)(frame + JANET_FRAME_SIZE) > fiber->stackstart ||
-            fiber->stackstart > fiber->stacktop ||
-            fiber->stacktop > fiber->maxstack) {
+    if ((int32_t)(frame + JANET_FRAME_SIZE) > fiber_stackstart ||
+            fiber_stackstart > fiber_stacktop ||
+            fiber_stacktop > fiber_maxstack) {
         janet_panic("fiber has incorrect stack setup");
     }
 
     /* Allocate stack memory */
-    fiber->capacity = fiber->stacktop + 10;
+    fiber->capacity = fiber_stacktop + 10;
     fiber->data = malloc(sizeof(Janet) * fiber->capacity);
     if (!fiber->data) {
         JANET_OUT_OF_MEMORY;
     }
+    for (int32_t i = 0; i < fiber->capacity; i++) {
+        fiber->data[i] = janet_wrap_nil();
+    }
 
     /* get frames */
-    stack = frame;
-    stacktop = fiber->stackstart - JANET_FRAME_SIZE;
+    int32_t stack = frame;
+    int32_t stacktop = fiber_stackstart - JANET_FRAME_SIZE;
     while (stack > 0) {
         JanetFunction *func = NULL;
         JanetFuncDef *def = NULL;
         JanetFuncEnv *env = NULL;
         int32_t frameflags = readint(st, &data);
-        int32_t prevframe = readint(st, &data);
-        int32_t pcdiff = readint(st, &data);
+        int32_t prevframe = readnat(st, &data);
+        int32_t pcdiff = readnat(st, &data);
 
         /* Get frame items */
         Janet *framestack = fiber->data + stack;
@@ -964,15 +988,7 @@ static const uint8_t *unmarshal_one_fiber(
         /* Check env */
         if (frameflags & JANET_STACKFRAME_HASENV) {
             frameflags &= ~JANET_STACKFRAME_HASENV;
-            int32_t offset = stack;
-            int32_t length = stacktop - stack;
             data = unmarshal_one_env(st, data, &env, flags + 1);
-            if (env->offset != 0 && env->offset != offset)
-                janet_panic("funcenv offset does not match fiber frame");
-            if (env->length != 0 && env->length != length)
-                janet_panic("funcenv length does not match fiber frame");
-            env->offset = offset;
-            env->length = length;
         }
 
         /* Error checking */
@@ -980,11 +996,11 @@ static const uint8_t *unmarshal_one_fiber(
         if (expected_framesize != stacktop - stack) {
             janet_panic("fiber stackframe size mismatch");
         }
-        if (pcdiff < 0 || pcdiff >= def->bytecode_length) {
+        if (pcdiff >= def->bytecode_length) {
             janet_panic("fiber stackframe has invalid pc");
         }
         if ((int32_t)(prevframe + JANET_FRAME_SIZE) > stack) {
-            janet_panic("fibre stackframe does not align with previous frame");
+            janet_panic("fiber stackframe does not align with previous frame");
         }
 
         /* Get stack items */
@@ -1007,25 +1023,32 @@ static const uint8_t *unmarshal_one_fiber(
     }
 
     /* Check for fiber env */
-    if (fiber->flags & JANET_FIBER_FLAG_HASENV) {
+    if (fiber_flags & JANET_FIBER_FLAG_HASENV) {
         Janet envv;
-        fiber->flags &= ~JANET_FIBER_FLAG_HASENV;
+        fiber_flags &= ~JANET_FIBER_FLAG_HASENV;
         data = unmarshal_one(st, data, &envv, flags + 1);
         janet_asserttype(envv, JANET_TABLE);
-        fiber->env = janet_unwrap_table(envv);
+        fiber_env = janet_unwrap_table(envv);
     }
 
     /* Check for child fiber */
-    if (fiber->flags & JANET_FIBER_FLAG_HASCHILD) {
+    if (fiber_flags & JANET_FIBER_FLAG_HASCHILD) {
         Janet fiberv;
-        fiber->flags &= ~JANET_FIBER_FLAG_HASCHILD;
+        fiber_flags &= ~JANET_FIBER_FLAG_HASCHILD;
         data = unmarshal_one(st, data, &fiberv, flags + 1);
         janet_asserttype(fiberv, JANET_FIBER);
         fiber->child = janet_unwrap_fiber(fiberv);
     }
 
-    /* Return data */
+    /* We have valid fiber, finally construct remaining fields. */
     fiber->frame = frame;
+    fiber->flags = fiber_flags;
+    fiber->stackstart = fiber_stackstart;
+    fiber->stacktop = fiber_stacktop;
+    fiber->maxstack = fiber_maxstack;
+    fiber->env = fiber_env;
+
+    /* Return data */
     *out = fiber;
     return data;
 }
@@ -1084,7 +1107,7 @@ static const uint8_t *unmarshal_one_abstract(UnmarshalState *st, const uint8_t *
     Janet key;
     data = unmarshal_one(st, data, &key, flags + 1);
     const JanetAbstractType *at = janet_get_abstract_type(key);
-    if (at == NULL) return NULL;
+    if (at == NULL) goto oops;
     if (at->unmarshal) {
         JanetMarshalContext context = {NULL, st, flags, data, at};
         *out = janet_wrap_abstract(at->unmarshal(&context));
@@ -1093,7 +1116,8 @@ static const uint8_t *unmarshal_one_abstract(UnmarshalState *st, const uint8_t *
         }
         return context.data;
     }
-    return NULL;
+oops:
+    janet_panic("invalid abstract type");
 }
 
 static const uint8_t *unmarshal_one(
@@ -1105,7 +1129,7 @@ static const uint8_t *unmarshal_one(
     MARSH_STACKCHECK;
     MARSH_EOS(st, data);
     lead = data[0];
-    if (lead < 200) {
+    if (lead < LB_REAL) {
         *out = janet_wrap_integer(readint(st, &data));
         return data;
     }
@@ -1159,7 +1183,7 @@ static const uint8_t *unmarshal_one(
         case LB_KEYWORD:
         case LB_REGISTRY: {
             data++;
-            int32_t len = readint(st, &data);
+            int32_t len = readnat(st, &data);
             MARSH_EOS(st, data - 1 + len);
             if (lead == LB_STRING) {
                 const uint8_t *str = janet_string(data, len);
@@ -1219,7 +1243,11 @@ static const uint8_t *unmarshal_one(
             /* Things that open with integers */
         {
             data++;
-            int32_t len = readint(st, &data);
+            int32_t len = readnat(st, &data);
+            /* DOS check */
+            if (lead != LB_REFERENCE) {
+                MARSH_EOS(st, data - 1 + len);
+            }
             if (lead == LB_ARRAY) {
                 /* Array */
                 JanetArray *array = janet_array(len);
@@ -1251,7 +1279,7 @@ static const uint8_t *unmarshal_one(
                 *out = janet_wrap_struct(janet_struct_end(struct_));
                 janet_v_push(st->lookup, *out);
             } else if (lead == LB_REFERENCE) {
-                if (len < 0 || len >= janet_v_count(st->lookup))
+                if (len >= janet_v_count(st->lookup))
                     janet_panicf("invalid reference %d", len);
                 *out = st->lookup[len];
             } else {
@@ -1274,6 +1302,42 @@ static const uint8_t *unmarshal_one(
             }
             return data;
         }
+        case LB_UNSAFE_POINTER: {
+            MARSH_EOS(st, data + sizeof(void *));
+            data++;
+            if (!(flags & JANET_MARSHAL_UNSAFE)) {
+                janet_panicf("unsafe flag not given, "
+                             "will not unmarshal raw pointer at index %d",
+                             (int)(data - st->start));
+            }
+            union {
+                void *ptr;
+                uint8_t bytes[sizeof(void *)];
+            } u;
+            memcpy(u.bytes, data, sizeof(void *));
+            data += sizeof(void *);
+            *out = janet_wrap_pointer(u.ptr);
+            janet_v_push(st->lookup, *out);
+            return data;
+        }
+        case LB_UNSAFE_CFUNCTION: {
+            MARSH_EOS(st, data + sizeof(JanetCFunction));
+            data++;
+            if (!(flags & JANET_MARSHAL_UNSAFE)) {
+                janet_panicf("unsafe flag not given, "
+                             "will not unmarshal function pointer at index %d",
+                             (int)(data - st->start));
+            }
+            union {
+                JanetCFunction ptr;
+                uint8_t bytes[sizeof(JanetCFunction)];
+            } u;
+            memcpy(u.bytes, data, sizeof(JanetCFunction));
+            data += sizeof(JanetCFunction);
+            *out = janet_wrap_cfunction(u.ptr);
+            janet_v_push(st->lookup, *out);
+            return data;
+        }
         default: {
             janet_panicf("unknown byte %x at index %d",
                          *data,
@@ -1344,17 +1408,17 @@ static const JanetReg marsh_cfuns[] = {
     {
         "marshal", cfun_marshal,
         JDOC("(marshal x &opt reverse-lookup buffer)\n\n"
-             "Marshal a janet value into a buffer and return the buffer. The buffer "
+             "Marshal a value into a buffer and return the buffer. The buffer "
              "can the later be unmarshalled to reconstruct the initial value. "
              "Optionally, one can pass in a reverse lookup table to not marshal "
              "aliased values that are found in the table. Then a forward"
-             "lookup table can be used to recover the original janet value when "
+             "lookup table can be used to recover the original value when "
              "unmarshalling.")
     },
     {
         "unmarshal", cfun_unmarshal,
         JDOC("(unmarshal buffer &opt lookup)\n\n"
-             "Unmarshal a janet value from a buffer. An optional lookup table "
+             "Unmarshal a value from a buffer. An optional lookup table "
              "can be provided to allow for aliases to be resolved. Returns the value "
              "unmarshalled from the buffer.")
     },

src/core/math.c

@@ -255,6 +255,10 @@ JANET_DEFINE_MATHOP(fabs, fabs)
 JANET_DEFINE_MATHOP(floor, floor)
 JANET_DEFINE_MATHOP(trunc, trunc)
 JANET_DEFINE_MATHOP(round, round)
+JANET_DEFINE_MATHOP(gamma, lgamma)
+JANET_DEFINE_MATHOP(log1p, log1p)
+JANET_DEFINE_MATHOP(erf, erf)
+JANET_DEFINE_MATHOP(erfc, erfc)
 
 #define JANET_DEFINE_MATH2OP(name, fop)\
 static Janet janet_##name(int32_t argc, Janet *argv) {\
@@ -267,6 +271,7 @@ static Janet janet_##name(int32_t argc, Janet *argv) {\
 JANET_DEFINE_MATH2OP(atan2, atan2)
 JANET_DEFINE_MATH2OP(pow, pow)
 JANET_DEFINE_MATH2OP(hypot, hypot)
+JANET_DEFINE_MATH2OP(nextafter, nextafter)
 
 static Janet janet_not(int32_t argc, Janet *argv) {
     janet_fixarity(argc, 1);
@@ -408,7 +413,7 @@ static const JanetReg math_cfuns[] = {
         "math/rng", cfun_rng_make,
         JDOC("(math/rng &opt seed)\n\n"
              "Creates a Psuedo-Random number generator, with an optional seed. "
-             "The seed should be an unsigned 32 bit integer. "
+             "The seed should be an unsigned 32 bit integer or a buffer. "
              "Do not use this for cryptography. Returns a core/rng abstract type.")
     },
     {
@@ -438,6 +443,26 @@ static const JanetReg math_cfuns[] = {
         JDOC("(math/exp2 x)\n\n"
              "Returns 2 to the power of x.")
     },
+    {
+        "math/log1p", janet_log1p,
+        JDOC("(math/log1p x)\n\n"
+             "Returns (log base e of x) + 1 more accurately than (+ (math/log x) 1)")
+    },
+    {
+        "math/gamma", janet_gamma,
+        JDOC("(math/gamma x)\n\n"
+             "Returns gamma(x).")
+    },
+    {
+        "math/erfc", janet_erfc,
+        JDOC("(math/erfc x)\n\n"
+             "Returns the complementary error function of x.")
+    },
+    {
+        "math/erf", janet_erf,
+        JDOC("(math/erf x)\n\n"
+             "Returns the error function of x.")
+    },
     {
         "math/expm1", janet_expm1,
         JDOC("(math/expm1 x)\n\n"
@@ -453,6 +478,11 @@ static const JanetReg math_cfuns[] = {
         JDOC("(math/round x)\n\n"
              "Returns the integer nearest to x.")
     },
+    {
+        "math/next", janet_nextafter,
+        JDOC("(math/next x y)\n\n"
+             "Returns the next representable floating point value after x in the direction of y.")
+    },
     {NULL, NULL, NULL}
 };
 
@@ -469,5 +499,19 @@ void janet_lib_math(JanetTable *env) {
               JDOC("The number representing positive infinity"));
     janet_def(env, "math/-inf", janet_wrap_number(-INFINITY),
               JDOC("The number representing negative infinity"));
+    janet_def(env, "math/int32-min", janet_wrap_number(INT32_MIN),
+              JDOC("The maximum contiguous integer representable by a 32 bit signed integer"));
+    janet_def(env, "math/int32-max", janet_wrap_number(INT32_MAX),
+              JDOC("The minimum contiguous integer represtenable by a 32 bit signed integer"));
+    janet_def(env, "math/int-min", janet_wrap_number(JANET_INTMIN_DOUBLE),
+              JDOC("The maximum contiguous integer representable by a double (2^53)"));
+    janet_def(env, "math/int-max", janet_wrap_number(JANET_INTMAX_DOUBLE),
+              JDOC("The minimum contiguous integer represtenable by a double (-(2^53))"));
+#ifdef NAN
+    janet_def(env, "math/nan", janet_wrap_number(NAN),
+#else
+    janet_def(env, "math/nan", janet_wrap_number(0.0 / 0.0),
+#endif
+              JDOC("Not a number (IEEE-754 NaN)"));
 #endif
 }

src/core/net.c

@@ -0,0 +1,682 @@
+/*
+* Copyright (c) 2020 Calvin Rose
+*
+* Permission is hereby granted, free of charge, to any person obtaining a copy
+* of this software and associated documentation files (the "Software"), to
+* deal in the Software without restriction, including without limitation the
+* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+* sell copies of the Software, and to permit persons to whom the Software is
+* furnished to do so, subject to the following conditions:
+*
+* The above copyright notice and this permission notice shall be included in
+* all copies or substantial portions of the Software.
+*
+* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+* IN THE SOFTWARE.
+*/
+
+#ifndef JANET_AMALG
+#include "features.h"
+#include <janet.h>
+#include "util.h"
+#endif
+
+#ifdef JANET_NET
+
+#ifdef JANET_WINDOWS
+#include <winsock2.h>
+#include <windows.h>
+#include <ws2tcpip.h>
+#pragma comment (lib, "Ws2_32.lib")
+#pragma comment (lib, "Mswsock.lib")
+#pragma comment (lib, "Advapi32.lib")
+#else
+#include <unistd.h>
+#include <signal.h>
+#include <sys/ioctl.h>
+#include <sys/types.h>
+#include <sys/socket.h>
+#include <poll.h>
+#include <netdb.h>
+#include <fcntl.h>
+#endif
+
+/*
+ * Streams
+ */
+
+#define JANET_STREAM_CLOSED 1
+#define JANET_STREAM_READABLE 2
+#define JANET_STREAM_WRITABLE 4
+
+static int janet_stream_close(void *p, size_t s);
+static int janet_stream_getter(void *p, Janet key, Janet *out);
+static const JanetAbstractType StreamAT = {
+    "core/stream",
+    janet_stream_close,
+    NULL,
+    janet_stream_getter,
+    JANET_ATEND_GET
+};
+
+#ifdef JANET_WINDOWS
+typedef struct {
+    SOCKET fd;
+    int flags;
+} JanetStream;
+#define JSOCKCLOSE(x) closesocket(x)
+#define JSOCKDEFAULT INVALID_SOCKET
+#define JLASTERR WSAGetLastError()
+#define JSOCKVALID(x) ((x) != INVALID_SOCKET)
+#define JEINTR WSAEINTR
+#define JEWOULDBLOCK WSAEWOULDBLOCK
+#define JEAGAIN WSAEWOULDBLOCK
+#define JPOLL WSAPoll
+#define JPollStruct WSAPOLLFD
+#define JSock SOCKET
+#define JReadInt long
+#define JSOCKFLAGS 0
+static JanetStream *make_stream(SOCKET fd, int flags) {
+    u_long iMode = 0;
+    JanetStream *stream = janet_abstract(&StreamAT, sizeof(JanetStream));
+    ioctlsocket(fd, FIONBIO, &iMode);
+    stream->fd = fd;
+    stream->flags = flags;
+    return stream;
+}
+#else
+typedef struct {
+    int fd;
+    int flags;
+} JanetStream;
+#define JSOCKCLOSE(x) close(x)
+#define JSOCKDEFAULT 0
+#define JLASTERR errno
+#define JSOCKVALID(x) ((x) >= 0)
+#define JEINTR EINTR
+#define JEWOULDBLOCK EWOULDBLOCK
+#define JEAGAIN EAGAIN
+#define JPOLL poll
+#define JPollStruct struct pollfd
+#define JSock int
+#define JReadInt ssize_t
+#ifdef SOCK_CLOEXEC
+#define JSOCKFLAGS SOCK_CLOEXEC
+#else
+#define JSOCKFLAGS 0
+#endif
+static JanetStream *make_stream(int fd, int flags) {
+    JanetStream *stream = janet_abstract(&StreamAT, sizeof(JanetStream));
+#if !defined(SOCK_CLOEXEC) && defined(O_CLOEXEC)
+    int extra = O_CLOEXEC;
+#else
+    int extra = 0;
+#endif
+    fcntl(fd, F_SETFL, fcntl(fd, F_GETFL, 0) | O_NONBLOCK | extra);
+    stream->fd = fd;
+    stream->flags = flags;
+    return stream;
+}
+#endif
+
+/* We pass this flag to all send calls to prevent sigpipe */
+#ifndef MSG_NOSIGNAL
+#define MSG_NOSIGNAL 0
+#endif
+
+static int janet_stream_close(void *p, size_t s) {
+    (void) s;
+    JanetStream *stream = p;
+    if (!(stream->flags & JANET_STREAM_CLOSED)) {
+        stream->flags |= JANET_STREAM_CLOSED;
+        JSOCKCLOSE(stream->fd);
+    }
+    return 0;
+}
+
+/*
+ * Event loop
+ */
+
+/* This large struct describes a waiting file descriptor, as well
+ * as what to do when we get an event for it. It is a variant type, where
+ * each variant implements a simple state machine. */
+typedef struct {
+
+    /* File descriptor to listen for events on. */
+    JanetStream *stream;
+
+    /* Fiber to resume when event finishes. Can be NULL, in which case,
+     * no fiber is resumed when event completes. */
+    JanetFiber *fiber;
+
+    /* What kind of event we are listening for.
+     * As more IO functionality get's added, we can
+     * expand this. */
+    enum {
+        JLE_READ_CHUNK,
+        JLE_READ_SOME,
+        JLE_READ_ACCEPT,
+        JLE_CONNECT,
+        JLE_WRITE_FROM_BUFFER,
+        JLE_WRITE_FROM_STRINGLIKE
+    } event_type;
+
+    /* Each variant can have a different payload. */
+    union {
+
+        /* JLE_READ_CHUNK/JLE_READ_SOME */
+        struct {
+            int32_t bytes_left;
+            JanetBuffer *buf;
+        } read_chunk;
+
+        /* JLE_READ_ACCEPT */
+        struct {
+            JanetFunction *handler;
+        } read_accept;
+
+        /* JLE_WRITE_FROM_BUFFER */
+        struct {
+            JanetBuffer *buf;
+            int32_t start;
+        } write_from_buffer;
+
+        /* JLE_WRITE_FROM_STRINGLIKE */
+        struct {
+            const uint8_t *str;
+            int32_t start;
+        } write_from_stringlike;
+
+    } data;
+
+} JanetLoopFD;
+
+#define JANET_LOOPFD_MAX 1024
+
+/* Global loop data */
+JANET_THREAD_LOCAL JPollStruct janet_vm_pollfds[JANET_LOOPFD_MAX];
+JANET_THREAD_LOCAL JanetLoopFD janet_vm_loopfds[JANET_LOOPFD_MAX];
+JANET_THREAD_LOCAL int janet_vm_loop_count;
+
+/* We could also add/remove gc roots. This is easier for now. */
+void janet_net_markloop(void) {
+    for (int i = 0; i < janet_vm_loop_count; i++) {
+        JanetLoopFD lfd = janet_vm_loopfds[i];
+        if (lfd.fiber != NULL) {
+            janet_mark(janet_wrap_fiber(lfd.fiber));
+        }
+        janet_mark(janet_wrap_abstract(lfd.stream));
+        switch (lfd.event_type) {
+            default:
+                break;
+            case JLE_READ_CHUNK:
+            case JLE_READ_SOME:
+                janet_mark(janet_wrap_buffer(lfd.data.read_chunk.buf));
+                break;
+            case JLE_READ_ACCEPT:
+                janet_mark(janet_wrap_function(lfd.data.read_accept.handler));
+                break;
+            case JLE_CONNECT:
+                break;
+            case JLE_WRITE_FROM_BUFFER:
+                janet_mark(janet_wrap_buffer(lfd.data.write_from_buffer.buf));
+                break;
+            case JLE_WRITE_FROM_STRINGLIKE:
+                janet_mark(janet_wrap_string(lfd.data.write_from_stringlike.str));
+        }
+    }
+}
+
+/* Add a loop fd to the global event loop */
+static int janet_loop_schedule(JanetLoopFD lfd, short events) {
+    if (janet_vm_loop_count == JANET_LOOPFD_MAX) {
+        return -1;
+    }
+    int index = janet_vm_loop_count++;
+    janet_vm_loopfds[index] = lfd;
+    janet_vm_pollfds[index].fd = lfd.stream->fd;
+    janet_vm_pollfds[index].events = events;
+    janet_vm_pollfds[index].revents = 0;
+    return index;
+}
+
+/* Remove event from list */
+static void janet_loop_rmindex(int index) {
+    janet_vm_loopfds[index] = janet_vm_loopfds[--janet_vm_loop_count];
+    janet_vm_pollfds[index] = janet_vm_pollfds[janet_vm_loop_count];
+}
+
+
+/* Return delta in number of loop fds. Abstracted out so
+ * we can separate out the polling logic */
+static size_t janet_loop_event(size_t index) {
+    JanetLoopFD *jlfd = janet_vm_loopfds + index;
+    JanetStream *stream = jlfd->stream;
+    JSock fd = stream->fd;
+    int ret = 1;
+    int should_resume = 0;
+    Janet resumeval = janet_wrap_nil();
+    if (stream->flags & JANET_STREAM_CLOSED) {
+        should_resume = 1;
+        ret = 0;
+    } else {
+        switch (jlfd->event_type) {
+            case JLE_READ_CHUNK:
+            case JLE_READ_SOME: {
+                JanetBuffer *buffer = jlfd->data.read_chunk.buf;
+                int32_t bytes_left = jlfd->data.read_chunk.bytes_left;
+                janet_buffer_extra(buffer, bytes_left);
+                if (!(stream->flags & JANET_STREAM_READABLE)) {
+                    should_resume = 1;
+                    ret = 0;
+                    break;
+                }
+                JReadInt nread;
+                do {
+                    nread = recv(fd, buffer->data + buffer->count, bytes_left, 0);
+                } while (nread == -1 && JLASTERR == JEINTR);
+                if (JLASTERR == JEAGAIN || JLASTERR == JEWOULDBLOCK) {
+                    ret = 1;
+                    break;
+                }
+
+                if (nread > 0) {
+                    buffer->count += nread;
+                    bytes_left -= nread;
+                } else {
+                    bytes_left = 0;
+                }
+                if (jlfd->event_type == JLE_READ_SOME || bytes_left == 0) {
+                    should_resume = 1;
+                    if (nread > 0) {
+                        resumeval = janet_wrap_buffer(buffer);
+                    }
+                    ret = 0;
+                } else {
+                    jlfd->data.read_chunk.bytes_left = bytes_left;
+                    ret = 1;
+                }
+                break;
+            }
+            case JLE_READ_ACCEPT: {
+                JSock connfd = accept(fd, NULL, NULL);
+                if (JSOCKVALID(connfd)) {
+                    /* Made a new connection socket */
+                    JanetStream *stream = make_stream(connfd, JANET_STREAM_READABLE | JANET_STREAM_WRITABLE);
+                    Janet streamv = janet_wrap_abstract(stream);
+                    JanetFunction *handler = jlfd->data.read_accept.handler;
+                    Janet out;
+                    JanetFiber *fiberp = NULL;
+                    /* Launch connection fiber */
+                    JanetSignal sig = janet_pcall(handler, 1, &streamv, &out, &fiberp);
+                    if (sig != JANET_SIGNAL_OK && sig != JANET_SIGNAL_EVENT) {
+                        janet_stacktrace(fiberp, out);
+                    }
+                }
+                ret = JANET_LOOPFD_MAX;
+                break;
+            }
+            case JLE_WRITE_FROM_BUFFER:
+            case JLE_WRITE_FROM_STRINGLIKE: {
+                int32_t start, len;
+                const uint8_t *bytes;
+                if (!(stream->flags & JANET_STREAM_WRITABLE)) {
+                    should_resume = 1;
+                    ret = 0;
+                    break;
+                }
+                if (jlfd->event_type == JLE_WRITE_FROM_BUFFER) {
+                    JanetBuffer *buffer = jlfd->data.write_from_buffer.buf;
+                    bytes = buffer->data;
+                    len = buffer->count;
+                    start = jlfd->data.write_from_buffer.start;
+                } else {
+                    bytes = jlfd->data.write_from_stringlike.str;
+                    len = janet_string_length(bytes);
+                    start = jlfd->data.write_from_stringlike.start;
+                }
+                if (start < len) {
+                    int32_t nbytes = len - start;
+                    JReadInt nwrote;
+                    do {
+                        nwrote = send(fd, bytes + start, nbytes, MSG_NOSIGNAL);
+                    } while (nwrote == -1 && JLASTERR == JEINTR);
+                    if (nwrote > 0) {
+                        start += nwrote;
+                    } else {
+                        start = len;
+                    }
+                }
+                if (start >= len) {
+                    should_resume = 1;
+                    ret = 0;
+                } else {
+                    if (jlfd->event_type == JLE_WRITE_FROM_BUFFER) {
+                        jlfd->data.write_from_buffer.start = start;
+                    } else {
+                        jlfd->data.write_from_stringlike.start = start;
+                    }
+                    ret = 1;
+                }
+                break;
+            }
+            case JLE_CONNECT: {
+                break;
+            }
+        }
+    }
+
+    /* Resume a fiber for some events */
+    if (NULL != jlfd->fiber && should_resume) {
+        /* Resume the fiber */
+        Janet out;
+        JanetSignal sig = janet_continue(jlfd->fiber, resumeval, &out);
+        if (sig != JANET_SIGNAL_OK && sig != JANET_SIGNAL_EVENT) {
+            janet_stacktrace(jlfd->fiber, out);
+        }
+    }
+
+    /* Remove this handler from the handler pool. */
+    if (should_resume) janet_loop_rmindex((int) index);
+
+    return ret;
+}
+
+static void janet_loop1(void) {
+    /* Remove closed file descriptors */
+    for (int i = 0; i < janet_vm_loop_count;) {
+        if (janet_vm_loopfds[i].stream->flags & JANET_STREAM_CLOSED) {
+            janet_loop_rmindex(i);
+        } else {
+            i++;
+        }
+    }
+    /* Poll */
+    if (janet_vm_loop_count == 0) return;
+    int ready;
+    do {
+        ready = JPOLL(janet_vm_pollfds, janet_vm_loop_count, -1);
+    } while (ready == -1 && JLASTERR == JEINTR);
+    if (ready == -1) return;
+    /* Handle events */
+    for (int i = 0; i < janet_vm_loop_count;) {
+        int revents = janet_vm_pollfds[i].revents;
+        janet_vm_pollfds[i].revents = 0;
+        if ((janet_vm_pollfds[i].events | POLLHUP | POLLERR) & revents) {
+            size_t delta = janet_loop_event(i);
+            i += (int) delta;
+        } else {
+            i++;
+        }
+    }
+}
+
+void janet_loop(void) {
+    while (janet_vm_loop_count) {
+        janet_loop1();
+    }
+}
+
+/*
+ * Scheduling Helpers
+ */
+
+#define JANET_SCHED_FSOME 1
+
+JANET_NO_RETURN static void janet_sched_read(JanetStream *stream, JanetBuffer *buf, int32_t nbytes, int flags) {
+    JanetLoopFD lfd;
+    lfd.stream = stream;
+    lfd.fiber = janet_root_fiber();
+    lfd.event_type = (flags & JANET_SCHED_FSOME) ? JLE_READ_SOME : JLE_READ_CHUNK;
+    lfd.data.read_chunk.buf = buf;
+    lfd.data.read_chunk.bytes_left = nbytes;
+    janet_loop_schedule(lfd, POLLIN);
+    janet_signalv(JANET_SIGNAL_EVENT, janet_wrap_nil());
+}
+
+JANET_NO_RETURN static void janet_sched_write_buffer(JanetStream *stream, JanetBuffer *buf) {
+    JanetLoopFD lfd;
+    lfd.stream = stream;
+    lfd.fiber = janet_root_fiber();
+    lfd.event_type = JLE_WRITE_FROM_BUFFER;
+    lfd.data.write_from_buffer.buf = buf;
+    lfd.data.write_from_buffer.start = 0;
+    janet_loop_schedule(lfd, POLLOUT);
+    janet_signalv(JANET_SIGNAL_EVENT, janet_wrap_nil());
+}
+
+JANET_NO_RETURN static void janet_sched_write_stringlike(JanetStream *stream, const uint8_t *str) {
+    JanetLoopFD lfd;
+    lfd.stream = stream;
+    lfd.fiber = janet_root_fiber();
+    lfd.event_type = JLE_WRITE_FROM_STRINGLIKE;
+    lfd.data.write_from_stringlike.str = str;
+    lfd.data.write_from_stringlike.start = 0;
+    janet_loop_schedule(lfd, POLLOUT);
+    janet_signalv(JANET_SIGNAL_EVENT, janet_wrap_nil());
+}
+
+/* Needs argc >= offset + 2 */
+static struct addrinfo *janet_get_addrinfo(Janet *argv, int32_t offset) {
+    /* Get host and port */
+    const char *host = janet_getcstring(argv, offset);
+    const char *port;
+    if (janet_checkint(argv[offset + 1])) {
+        port = (const char *)janet_to_string(argv[offset + 1]);
+    } else {
+        port = janet_getcstring(argv, offset + 1);
+    }
+    /* getaddrinfo */
+    struct addrinfo *ai = NULL;
+    struct addrinfo hints;
+    memset(&hints, 0, sizeof(hints));
+    hints.ai_family = AF_UNSPEC;
+    hints.ai_socktype = SOCK_STREAM;
+    hints.ai_protocol = 0;
+    hints.ai_flags = AI_PASSIVE;
+    int status = getaddrinfo(host, port, &hints, &ai);
+    if (status) {
+        janet_panicf("could not get address info: %s", gai_strerror(status));
+    }
+    return ai;
+}
+
+/*
+ * C Funs
+ */
+
+static Janet cfun_net_connect(int32_t argc, Janet *argv) {
+    janet_fixarity(argc, 2);
+
+    struct addrinfo *ai = janet_get_addrinfo(argv, 0);
+
+    /* Create socket */
+    JSock sock = socket(ai->ai_family, ai->ai_socktype | JSOCKFLAGS, ai->ai_protocol);
+    if (!JSOCKVALID(sock)) {
+        freeaddrinfo(ai);
+        janet_panic("could not create socket");
+    }
+
+    /* Connect to socket */
+    int status = connect(sock, ai->ai_addr, (int) ai->ai_addrlen);
+    freeaddrinfo(ai);
+    if (status == -1) {
+        JSOCKCLOSE(sock);
+        janet_panic("could not connect to socket");
+    }
+
+    /* Wrap socket in abstract type JanetStream */
+    JanetStream *stream = make_stream(sock, JANET_STREAM_READABLE | JANET_STREAM_WRITABLE);
+    return janet_wrap_abstract(stream);
+}
+
+static Janet cfun_net_server(int32_t argc, Janet *argv) {
+    janet_fixarity(argc, 3);
+
+    /* Get host, port, and handler*/
+    JanetFunction *fun = janet_getfunction(argv, 2);
+
+    struct addrinfo *ai = janet_get_addrinfo(argv, 0);
+
+    /* Check all addrinfos in a loop for the first that we can bind to. */
+    JSock sfd = JSOCKDEFAULT;
+    struct addrinfo *rp = NULL;
+    for (rp = ai; rp != NULL; rp = rp->ai_next) {
+        sfd = socket(rp->ai_family, rp->ai_socktype | JSOCKFLAGS, rp->ai_protocol);
+        if (!JSOCKVALID(sfd)) continue;
+        /* Set various socket options */
+        int enable = 1;
+        if (setsockopt(sfd, SOL_SOCKET, SO_REUSEADDR, (char *) &enable, sizeof(int)) < 0) {
+            JSOCKCLOSE(sfd);
+            janet_panic("setsockopt(SO_REUSEADDR) failed");
+        }
+#ifdef SO_NOSIGPIPE
+        if (setsockopt(sfd, SOL_SOCKET, SO_NOSIGPIPE, &enable, sizeof(int)) < 0) {
+            JSOCKCLOSE(sfd);
+            janet_panic("setsockopt(SO_NOSIGPIPE) failed");
+        }
+#endif
+#ifdef SO_REUSEPORT
+        if (setsockopt(sfd, SOL_SOCKET, SO_REUSEPORT, &enable, sizeof(int)) < 0) {
+            JSOCKCLOSE(sfd);
+            janet_panic("setsockopt(SO_REUSEPORT) failed");
+        }
+#endif
+        /* Bind */
+        if (bind(sfd, rp->ai_addr, (int) rp->ai_addrlen) == 0) break;
+        JSOCKCLOSE(sfd);
+    }
+    if (NULL == rp) {
+        freeaddrinfo(ai);
+        janet_panic("could not bind to any sockets");
+    }
+
+    /* listen */
+    int status = listen(sfd, 1024);
+    freeaddrinfo(ai);
+    if (status) {
+        JSOCKCLOSE(sfd);
+        janet_panic("could not listen on file descriptor");
+    }
+
+    /* Put sfd on our loop */
+    JanetLoopFD lfd;
+    memset(&lfd, 0, sizeof(lfd));
+    lfd.stream = make_stream(sfd, 0);
+    lfd.event_type = JLE_READ_ACCEPT;
+    lfd.data.read_accept.handler = fun;
+    janet_loop_schedule(lfd, POLLIN);
+
+    return janet_wrap_abstract(lfd.stream);
+}
+
+static Janet cfun_stream_read(int32_t argc, Janet *argv) {
+    janet_arity(argc, 2, 3);
+    JanetStream *stream = janet_getabstract(argv, 0, &StreamAT);
+    int32_t n = janet_getnat(argv, 1);
+    JanetBuffer *buffer = janet_optbuffer(argv, argc, 2, 10);
+    janet_sched_read(stream, buffer, n, JANET_SCHED_FSOME);
+}
+
+static Janet cfun_stream_chunk(int32_t argc, Janet *argv) {
+    janet_arity(argc, 2, 3);
+    JanetStream *stream = janet_getabstract(argv, 0, &StreamAT);
+    int32_t n = janet_getnat(argv, 1);
+    JanetBuffer *buffer = janet_optbuffer(argv, argc, 2, 10);
+    janet_sched_read(stream, buffer, n, 0);
+}
+
+static Janet cfun_stream_close(int32_t argc, Janet *argv) {
+    janet_fixarity(argc, 1);
+    JanetStream *stream = janet_getabstract(argv, 0, &StreamAT);
+    janet_stream_close(stream, 0);
+    return janet_wrap_nil();
+}
+
+static Janet cfun_stream_write(int32_t argc, Janet *argv) {
+    janet_fixarity(argc, 2);
+    JanetStream *stream = janet_getabstract(argv, 0, &StreamAT);
+    if (janet_checktype(argv[1], JANET_BUFFER)) {
+        janet_sched_write_buffer(stream, janet_getbuffer(argv, 1));
+    } else {
+        JanetByteView bytes = janet_getbytes(argv, 1);
+        janet_sched_write_stringlike(stream, bytes.bytes);
+    }
+}
+
+static const JanetMethod stream_methods[] = {
+    {"chunk", cfun_stream_chunk},
+    {"close", cfun_stream_close},
+    {"read", cfun_stream_read},
+    {"write", cfun_stream_write},
+    {NULL, NULL}
+};
+
+static int janet_stream_getter(void *p, Janet key, Janet *out) {
+    (void) p;
+    if (!janet_checktype(key, JANET_KEYWORD)) return 0;
+    return janet_getmethod(janet_unwrap_keyword(key), stream_methods, out);
+}
+
+static const JanetReg net_cfuns[] = {
+    {
+        "net/server", cfun_net_server,
+        JDOC("(net/server host port handler)\n\n"
+             "Start a TCP server. handler is a function that will be called with a stream "
+             "on each connection to the server. Returns a new stream that is neither readable nor "
+             "writeable.")
+    },
+    {
+        "net/read", cfun_stream_read,
+        JDOC("(net/read stream nbytes &opt buf)\n\n"
+             "Read up to n bytes from a stream, suspending the current fiber until the bytes are available. "
+             "If less than n bytes are available (and more than 0), will push those bytes and return early. "
+             "Returns a buffer with up to n more bytes in it.")
+    },
+    {
+        "net/chunk", cfun_stream_chunk,
+        JDOC("(net/chunk stream nbytes &opt buf)\n\n"
+             "Same a net/read, but will wait for all n bytes to arrive rather than return early.")
+    },
+    {
+        "net/write", cfun_stream_write,
+        JDOC("(net/write stream data)\n\n"
+             "Write data to a stream, suspending the current fiber until the write "
+             "completes. Returns stream.")
+    },
+    {
+        "net/close", cfun_stream_close,
+        JDOC("(net/close stream)\n\n"
+             "Close a stream so that no further communication can occur.")
+    },
+    {
+        "net/connect", cfun_net_connect,
+        JDOC("(net/connect host port)\n\n"
+             "Open a connection to communicate with a server. Returns a duplex stream "
+             "that can be used to communicate with the server.")
+    },
+    {NULL, NULL, NULL}
+};
+
+void janet_lib_net(JanetTable *env) {
+    janet_vm_loop_count = 0;
+#ifdef JANET_WINDOWS
+    WSADATA wsaData;
+    janet_assert(!WSAStartup(MAKEWORD(2, 2), &wsaData), "could not start winsock");
+#endif
+    janet_core_cfuns(env, NULL, net_cfuns);
+}
+
+void janet_net_deinit(void) {
+#ifdef JANET_WINDOWS
+    WSACleanup();
+#endif
+}
+
+#endif

src/core/os.c

@@ -39,6 +39,10 @@
 
 #define RETRY_EINTR(RC, CALL) do { (RC) = CALL; } while((RC) < 0 && errno == EINTR)
 
+#ifdef JANET_APPLE
+#include <AvailabilityMacros.h>
+#endif
+
 #ifdef JANET_WINDOWS
 #include <windows.h>
 #include <direct.h>
@@ -66,7 +70,7 @@ extern char **environ;
 
 /* Setting C99 standard makes this not available, but it should
  * work/link properly if we detect a BSD */
-#if defined(JANET_BSD) || defined(JANET_APPLE)
+#if defined(JANET_BSD) || defined(MAC_OS_X_VERSION_10_7)
 void arc4random_buf(void *buf, size_t nbytes);
 #endif
 
@@ -159,6 +163,8 @@ static Janet os_arch(int32_t argc, Janet *argv) {
     return janet_ckeywordv("arm");
 #elif (defined(__sparc__))
     return janet_ckeywordv("sparc");
+#elif (defined(__ppc__))
+    return janet_ckeywordv("ppc");
 #else
     return janet_ckeywordv("unknown");
 #endif
@@ -182,18 +188,9 @@ static Janet os_exit(int32_t argc, Janet *argv) {
     return janet_wrap_nil();
 }
 
-#ifdef JANET_REDUCED_OS
-/* Provide a dud os/getenv so boot.janet and init.janet work, but nothing else */
-
-static Janet os_getenv(int32_t argc, Janet *argv) {
-    (void) argv;
-    janet_arity(argc, 1, 2);
-    return janet_wrap_nil();
-}
-
-#else
-/* Provide full os functionality */
+#ifndef JANET_REDUCED_OS
 
+#ifndef JANET_NO_PROCESSES
 /* Get env for os_execute */
 static char **os_execute_env(int32_t argc, const Janet *argv) {
     char **envp = NULL;
@@ -348,7 +345,7 @@ static Janet os_execute(int32_t argc, Janet *argv) {
 
     JanetBuffer *buf = os_exec_escape(exargs);
     if (buf->count > 8191) {
-        janet_panic("command line string too long");
+        janet_panic("command line string too long (max 8191 characters)");
     }
     const char *path = (const char *) janet_unwrap_string(exargs.items[0]);
     char *cargv[2] = {(char *) buf->data, NULL};
@@ -389,15 +386,26 @@ static Janet os_execute(int32_t argc, Janet *argv) {
     char *const *cargv = (char *const *)child_argv;
 
     /* Use posix_spawn to spawn new process */
+
+    int use_environ = !janet_flag_at(flags, 0);
+
+    if (use_environ) {
+        janet_lock_environ();
+    }
+
     pid_t pid;
     if (janet_flag_at(flags, 1)) {
         status = posix_spawnp(&pid,
                               child_argv[0], NULL, NULL, cargv,
-                              janet_flag_at(flags, 0) ? envp : environ);
+                              use_environ ? environ : envp);
     } else {
         status = posix_spawn(&pid,
                              child_argv[0], NULL, NULL, cargv,
-                             janet_flag_at(flags, 0) ? envp : environ);
+                             use_environ ? environ : envp);
+    }
+
+    if (use_environ) {
+        janet_unlock_environ();
     }
 
     /* Wait for child */
@@ -433,6 +441,8 @@ static Janet os_shell(int32_t argc, Janet *argv) {
            : janet_wrap_boolean(stat);
 }
 
+#endif /* JANET_NO_PROCESSES */
+
 static Janet os_environ(int32_t argc, Janet *argv) {
     (void) argv;
     janet_fixarity(argc, 0);
@@ -504,39 +514,11 @@ static Janet os_time(int32_t argc, Janet *argv) {
     return janet_wrap_number(dtime);
 }
 
-/* Clock shims */
-#ifdef JANET_WINDOWS
-static int gettime(struct timespec *spec) {
-    FILETIME ftime;
-    GetSystemTimeAsFileTime(&ftime);
-    int64_t wintime = (int64_t)(ftime.dwLowDateTime) | ((int64_t)(ftime.dwHighDateTime) << 32);
-    /* Windows epoch is January 1, 1601 apparently */
-    wintime -= 116444736000000000LL;
-    spec->tv_sec  = wintime / 10000000LL;
-    /* Resolution is 100 nanoseconds. */
-    spec->tv_nsec = wintime % 10000000LL * 100;
-    return 0;
-}
-#elif defined(__MACH__)
-static int gettime(struct timespec *spec) {
-    clock_serv_t cclock;
-    mach_timespec_t mts;
-    host_get_clock_service(mach_host_self(), CALENDAR_CLOCK, &cclock);
-    clock_get_time(cclock, &mts);
-    mach_port_deallocate(mach_task_self(), cclock);
-    spec->tv_sec = mts.tv_sec;
-    spec->tv_nsec = mts.tv_nsec;
-    return 0;
-}
-#else
-#define gettime(TV) clock_gettime(CLOCK_MONOTONIC, (TV))
-#endif
-
 static Janet os_clock(int32_t argc, Janet *argv) {
     janet_fixarity(argc, 0);
     (void) argv;
     struct timespec tv;
-    if (gettime(&tv)) janet_panic("could not get time");
+    if (janet_gettime(&tv)) janet_panic("could not get time");
     double dtime = tv.tv_sec + (tv.tv_nsec / 1E9);
     return janet_wrap_number(dtime);
 }
@@ -600,13 +582,14 @@ static Janet os_cryptorand(int32_t argc, Janet *argv) {
             v = v >> 8;
         }
     }
-#elif defined(JANET_LINUX)
+#elif defined(JANET_LINUX) || ( defined(JANET_APPLE) && !defined(MAC_OS_X_VERSION_10_7) )
     /* We should be able to call getrandom on linux, but it doesn't seem
        to be uniformly supported on linux distros.
-       In both cases, use this fallback path for now... */
+       On Mac, arc4random_buf wasn't available on until 10.7.
+       In these cases, use this fallback path for now... */
     int rc;
     int randfd;
-    RETRY_EINTR(randfd, open("/dev/urandom", O_RDONLY));
+    RETRY_EINTR(randfd, open("/dev/urandom", O_RDONLY | O_CLOEXEC));
     if (randfd < 0)
         janet_panic(genericerr);
     while (n > 0) {
@@ -620,7 +603,7 @@ static Janet os_cryptorand(int32_t argc, Janet *argv) {
         n -= nread;
     }
     RETRY_EINTR(rc, close(randfd));
-#elif defined(JANET_BSD) || defined(JANET_APPLE)
+#elif defined(JANET_BSD) || defined(MAC_OS_X_VERSION_10_7)
     (void) genericerr;
     arc4random_buf(buffer->data + offset, n);
 #else
@@ -731,7 +714,10 @@ static timeint_t entry_getint(Janet env_entry, char *field) {
 static Janet os_mktime(int32_t argc, Janet *argv) {
     janet_arity(argc, 1, 2);
     time_t t;
-    struct tm t_info = { 0 };
+    struct tm t_info;
+
+    /* Use memset instead of = {0} to silence paranoid warning in macos */
+    memset(&t_info, 0, sizeof(t_info));
 
     if (!janet_checktype(argv[0], JANET_TABLE) &&
             !janet_checktype(argv[0], JANET_STRUCT))
@@ -750,8 +736,8 @@ static Janet os_mktime(int32_t argc, Janet *argv) {
         t = mktime(&t_info);
     } else {
         /* utc time */
-#ifdef __sun
-        janet_panic("os/mktime UTC not supported on Solaris");
+#ifdef JANET_NO_UTC_MKTIME
+        janet_panic("os/mktime UTC not supported on this platform");
         return janet_wrap_nil();
 #else
         t = timegm(&t_info);
@@ -765,6 +751,12 @@ static Janet os_mktime(int32_t argc, Janet *argv) {
     return janet_wrap_number((double)t);
 }
 
+#ifdef JANET_NO_SYMLINKS
+#define j_symlink link
+#else
+#define j_symlink symlink
+#endif
+
 static Janet os_link(int32_t argc, Janet *argv) {
     janet_arity(argc, 2, 3);
 #ifdef JANET_WINDOWS
@@ -775,7 +767,7 @@ static Janet os_link(int32_t argc, Janet *argv) {
 #else
     const char *oldpath = janet_getcstring(argv, 0);
     const char *newpath = janet_getcstring(argv, 1);
-    int res = ((argc == 3 && janet_truthy(argv[2])) ? symlink : link)(oldpath, newpath);
+    int res = ((argc == 3 && janet_truthy(argv[2])) ? j_symlink : link)(oldpath, newpath);
     if (-1 == res) janet_panicf("%s: %s -> %s", strerror(errno), oldpath, newpath);
     return janet_wrap_nil();
 #endif
@@ -791,12 +783,14 @@ static Janet os_symlink(int32_t argc, Janet *argv) {
 #else
     const char *oldpath = janet_getcstring(argv, 0);
     const char *newpath = janet_getcstring(argv, 1);
-    int res = symlink(oldpath, newpath);
+    int res = j_symlink(oldpath, newpath);
     if (-1 == res) janet_panicf("%s: %s -> %s", strerror(errno), oldpath, newpath);
     return janet_wrap_nil();
 #endif
 }
 
+#undef j_symlink
+
 static Janet os_mkdir(int32_t argc, Janet *argv) {
     janet_fixarity(argc, 1);
     const char *path = janet_getcstring(argv, 0);
@@ -862,6 +856,7 @@ static Janet os_remove(int32_t argc, Janet *argv) {
     return janet_wrap_nil();
 }
 
+#ifndef JANET_NO_SYMLINKS
 static Janet os_readlink(int32_t argc, Janet *argv) {
     janet_fixarity(argc, 1);
 #ifdef JANET_WINDOWS
@@ -878,42 +873,26 @@ static Janet os_readlink(int32_t argc, Janet *argv) {
     return janet_stringv((const uint8_t *)buffer, len);
 #endif
 }
+#endif
 
 #ifdef JANET_WINDOWS
-static const uint8_t *janet_decode_permissions(unsigned short m) {
-    uint8_t flags[9] = {0};
-    flags[0] = flags[3] = flags[6] = (m & S_IREAD) ? 'r' : '-';
-    flags[1] = flags[4] = flags[7] = (m & S_IWRITE) ? 'w' : '-';
-    flags[2] = flags[5] = flags[8] = (m & S_IEXEC) ? 'x' : '-';
-    return janet_string(flags, sizeof(flags));
+
+typedef struct _stat jstat_t;
+typedef unsigned short jmode_t;
+
+static int32_t janet_perm_to_unix(unsigned short m) {
+    int32_t ret = 0;
+    if (m & S_IEXEC) ret |= 0111;
+    if (m & S_IWRITE) ret |= 0222;
+    if (m & S_IREAD) ret |= 0444;
+    return ret;
 }
 
-static unsigned short janet_encode_permissions(Janet *argv, int32_t n) {
-    if (janet_checkint(argv[n])) {
-        int32_t x = janet_unwrap_integer(argv[n]);
-        if (x < 0 || x > 0777) {
-            janet_panicf("expected integer in range [0, 8r777], got %v", argv[n]);
-        }
-        unsigned short m = 0;
-        if (x & 1 || x & 010 || x & 0100) m |= S_IEXEC;
-        if (x & 2 || x & 020 || x & 0200) m |= S_IWRITE;
-        if (x & 4 || x & 040 || x & 0400) m |= S_IREAD;
-        return m;
-    }
-    JanetString perm = janet_getstring(argv, n);
-    if (janet_string_length(perm) != 9) {
-        janet_panicf("expected string of length 9, got %S", perm);
-    }
+static unsigned short janet_perm_from_unix(int32_t x) {
     unsigned short m = 0;
-    if (perm[0] == 'r') m |= S_IREAD;
-    if (perm[1] == 'w') m |= S_IWRITE;
-    if (perm[2] == 'x') m |= S_IEXEC;
-    if (perm[3] == 'r') m |= S_IREAD;
-    if (perm[4] == 'w') m |= S_IWRITE;
-    if (perm[5] == 'x') m |= S_IEXEC;
-    if (perm[6] == 'r') m |= S_IREAD;
-    if (perm[7] == 'w') m |= S_IWRITE;
-    if (perm[8] == 'x') m |= S_IEXEC;
+    if (x & 111) m |= S_IEXEC;
+    if (x & 222) m |= S_IWRITE;
+    if (x & 444) m |= S_IREAD;
     return m;
 }
 
@@ -924,44 +903,22 @@ static const uint8_t *janet_decode_mode(unsigned short m) {
     else if (m & _S_IFCHR) str = "character";
     return janet_ckeyword(str);
 }
-#else
-static const uint8_t *janet_decode_permissions(mode_t m) {
-    uint8_t flags[9] = {0};
-    flags[0] = (m & S_IRUSR) ? 'r' : '-';
-    flags[1] = (m & S_IWUSR) ? 'w' : '-';
-    flags[2] = (m & S_IXUSR) ? 'x' : '-';
-    flags[3] = (m & S_IRGRP) ? 'r' : '-';
-    flags[4] = (m & S_IWGRP) ? 'w' : '-';
-    flags[5] = (m & S_IXGRP) ? 'x' : '-';
-    flags[6] = (m & S_IROTH) ? 'r' : '-';
-    flags[7] = (m & S_IWOTH) ? 'w' : '-';
-    flags[8] = (m & S_IXOTH) ? 'x' : '-';
-    return janet_string(flags, sizeof(flags));
+
+static int32_t janet_decode_permissions(jmode_t mode) {
+    return (int32_t)(mode & (S_IEXEC | S_IWRITE | S_IREAD));
 }
 
-static mode_t janet_encode_permissions(Janet *argv, int32_t n) {
-    if (janet_checkint(argv[n])) {
-        int32_t x = janet_unwrap_integer(argv[n]);
-        if (x < 0 || x > 0777) {
-            janet_panicf("expected integer in range [0, 8r777], got %v", argv[n]);
-        }
-        return (mode_t) x;
-    }
-    JanetString perm = janet_getstring(argv, n);
-    if (janet_string_length(perm) != 9) {
-        janet_panicf("expected string of length 9, got %S", perm);
-    }
-    mode_t m = 0;
-    if (perm[0] == 'r') m |= S_IRUSR;
-    if (perm[1] == 'w') m |= S_IWUSR;
-    if (perm[2] == 'x') m |= S_IXUSR;
-    if (perm[3] == 'r') m |= S_IRGRP;
-    if (perm[4] == 'w') m |= S_IWGRP;
-    if (perm[5] == 'x') m |= S_IXGRP;
-    if (perm[6] == 'r') m |= S_IROTH;
-    if (perm[7] == 'w') m |= S_IWOTH;
-    if (perm[8] == 'x') m |= S_IXOTH;
-    return m;
+#else
+
+typedef struct stat jstat_t;
+typedef mode_t jmode_t;
+
+static int32_t janet_perm_to_unix(mode_t m) {
+    return (int32_t) m;
+}
+
+static mode_t janet_perm_from_unix(int32_t x) {
+    return (mode_t) x;
 }
 
 static const uint8_t *janet_decode_mode(mode_t m) {
@@ -975,13 +932,64 @@ static const uint8_t *janet_decode_mode(mode_t m) {
     else if (S_ISCHR(m)) str = "character";
     return janet_ckeyword(str);
 }
+
+static int32_t janet_decode_permissions(jmode_t mode) {
+    return (int32_t)(mode & 0777);
+}
+
 #endif
 
-#ifdef JANET_WINDOWS
-typedef struct _stat jstat_t;
-#else
-typedef struct stat jstat_t;
-#endif
+static int32_t os_parse_permstring(const uint8_t *perm) {
+    int32_t m = 0;
+    if (perm[0] == 'r') m |= 0400;
+    if (perm[1] == 'w') m |= 0200;
+    if (perm[2] == 'x') m |= 0100;
+    if (perm[3] == 'r') m |= 0040;
+    if (perm[4] == 'w') m |= 0020;
+    if (perm[5] == 'x') m |= 0010;
+    if (perm[6] == 'r') m |= 0004;
+    if (perm[7] == 'w') m |= 0002;
+    if (perm[8] == 'x') m |= 0001;
+    return m;
+}
+
+static Janet os_make_permstring(int32_t permissions) {
+    uint8_t bytes[9] = {0};
+    bytes[0] = (permissions & 0400) ? 'r' : '-';
+    bytes[1] = (permissions & 0200) ? 'w' : '-';
+    bytes[2] = (permissions & 0100) ? 'x' : '-';
+    bytes[3] = (permissions & 0040) ? 'r' : '-';
+    bytes[4] = (permissions & 0020) ? 'w' : '-';
+    bytes[5] = (permissions & 0010) ? 'x' : '-';
+    bytes[6] = (permissions & 0004) ? 'r' : '-';
+    bytes[7] = (permissions & 0002) ? 'w' : '-';
+    bytes[8] = (permissions & 0001) ? 'x' : '-';
+    return janet_stringv(bytes, sizeof(bytes));
+}
+
+static int32_t os_get_unix_mode(const Janet *argv, int32_t n) {
+    int32_t unix_mode;
+    if (janet_checkint(argv[n])) {
+        /* Integer mode */
+        int32_t x = janet_unwrap_integer(argv[n]);
+        if (x < 0 || x > 0777) {
+            janet_panicf("bad slot #%d, expected integer in range [0, 8r777], got %v", n, argv[n]);
+        }
+        unix_mode = x;
+    } else {
+        /* Bytes mode */
+        JanetByteView bytes = janet_getbytes(argv, n);
+        if (bytes.len != 9) {
+            janet_panicf("bad slot #%d: expected byte sequence of length 9, got %v", n, argv[n]);
+        }
+        unix_mode = os_parse_permstring(bytes.bytes);
+    }
+    return unix_mode;
+}
+
+static jmode_t os_getmode(const Janet *argv, int32_t n) {
+    return janet_perm_from_unix(os_get_unix_mode(argv, n));
+}
 
 /* Getters */
 static Janet os_stat_dev(jstat_t *st) {
@@ -993,8 +1001,11 @@ static Janet os_stat_inode(jstat_t *st) {
 static Janet os_stat_mode(jstat_t *st) {
     return janet_wrap_keyword(janet_decode_mode(st->st_mode));
 }
+static Janet os_stat_int_permissions(jstat_t *st) {
+    return janet_wrap_integer(janet_perm_to_unix(janet_decode_permissions(st->st_mode)));
+}
 static Janet os_stat_permissions(jstat_t *st) {
-    return janet_wrap_string(janet_decode_permissions(st->st_mode));
+    return os_make_permstring(janet_perm_to_unix(janet_decode_permissions(st->st_mode)));
 }
 static Janet os_stat_uid(jstat_t *st) {
     return janet_wrap_number(st->st_uid);
@@ -1045,6 +1056,7 @@ static const struct OsStatGetter os_stat_getters[] = {
     {"dev", os_stat_dev},
     {"inode", os_stat_inode},
     {"mode", os_stat_mode},
+    {"int-permissions", os_stat_int_permissions},
     {"permissions", os_stat_permissions},
     {"uid", os_stat_uid},
     {"gid", os_stat_gid},
@@ -1122,14 +1134,27 @@ static Janet os_chmod(int32_t argc, Janet *argv) {
     janet_fixarity(argc, 2);
     const char *path = janet_getcstring(argv, 0);
 #ifdef JANET_WINDOWS
-    int res = _chmod(path, janet_encode_permissions(argv, 1));
+    int res = _chmod(path, os_getmode(argv, 1));
 #else
-    int res = chmod(path, janet_encode_permissions(argv, 1));
+    int res = chmod(path, os_getmode(argv, 1));
 #endif
     if (-1 == res) janet_panicf("%s: %s", strerror(errno), path);
     return janet_wrap_nil();
 }
 
+#ifndef JANET_NO_UMASK
+static Janet os_umask(int32_t argc, Janet *argv) {
+    janet_fixarity(argc, 1);
+    int mask = (int) os_getmode(argv, 0);
+#ifdef JANET_WINDOWS
+    int res = _umask(mask);
+#else
+    int res = umask(mask);
+#endif
+    return janet_wrap_integer(janet_perm_to_unix(res));
+}
+#endif
+
 static Janet os_dir(int32_t argc, Janet *argv) {
     janet_arity(argc, 1, 2);
     const char *dir = janet_getcstring(argv, 0);
@@ -1178,12 +1203,15 @@ static Janet os_rename(int32_t argc, Janet *argv) {
 
 static Janet os_realpath(int32_t argc, Janet *argv) {
     janet_fixarity(argc, 1);
-#ifdef JANET_WINDOWS
-    (void) argv;
-    janet_panic("os/realpath not supported on Windows");
-#else
     const char *src = janet_getcstring(argv, 0);
+#ifdef JANET_NO_REALPATH
+    janet_panic("os/realpath not enabled for this platform");
+#else
+#ifdef JANET_WINDOWS
+    char *dest = _fullpath(NULL, src, _MAX_PATH);
+#else
     char *dest = realpath(src, NULL);
+#endif
     if (NULL == dest) janet_panicf("%s: %s", strerror(errno), src);
     Janet ret = janet_cstringv(dest);
     free(dest);
@@ -1191,6 +1219,16 @@ static Janet os_realpath(int32_t argc, Janet *argv) {
 #endif
 }
 
+static Janet os_permission_string(int32_t argc, Janet *argv) {
+    janet_fixarity(argc, 1);
+    return os_make_permstring(os_get_unix_mode(argv, 0));
+}
+
+static Janet os_permission_int(int32_t argc, Janet *argv) {
+    janet_fixarity(argc, 1);
+    return janet_wrap_integer(os_get_unix_mode(argv, 0));
+}
+
 #endif /* JANET_REDUCED_OS */
 
 static const JanetReg os_cfuns[] = {
@@ -1211,12 +1249,8 @@ static const JanetReg os_cfuns[] = {
              "\t:freebsd\n"
              "\t:openbsd\n"
              "\t:netbsd\n"
-             "\t:posix - A POSIX compatible system (default)")
-    },
-    {
-        "os/getenv", os_getenv,
-        JDOC("(os/getenv variable &opt dflt)\n\n"
-             "Get the string value of an environment variable.")
+             "\t:posix - A POSIX compatible system (default)\n\n"
+             "May also return a custom keyword specified at build time.")
     },
     {
         "os/arch", os_arch,
@@ -1236,11 +1270,16 @@ static const JanetReg os_cfuns[] = {
         JDOC("(os/environ)\n\n"
              "Get a copy of the os environment table.")
     },
+    {
+        "os/getenv", os_getenv,
+        JDOC("(os/getenv variable &opt dflt)\n\n"
+             "Get the string value of an environment variable.")
+    },
     {
         "os/dir", os_dir,
         JDOC("(os/dir dir &opt array)\n\n"
              "Iterate over files and subdirectories in a directory. Returns an array of paths parts, "
-             "with only the filename or directory name and no prefix.")
+             "with only the file name or directory name and no prefix.")
     },
     {
         "os/stat", os_stat,
@@ -1249,7 +1288,8 @@ static const JanetReg os_cfuns[] = {
              " only that information from stat. If the file or directory does not exist, returns nil. The keys are\n\n"
              "\t:dev - the device that the file is on\n"
              "\t:mode - the type of file, one of :file, :directory, :block, :character, :fifo, :socket, :link, or :other\n"
-             "\t:permissions - A unix permission string like \"rwx--x--x\". On windows, a string like \"rwx\".\n"
+             "\t:int-permissions - A Unix permission integer like 8r744\n"
+             "\t:permissions - A Unix permission string like \"rwxr--r--\"\n"
              "\t:uid - File uid\n"
              "\t:gid - File gid\n"
              "\t:nlink - number of links to file\n"
@@ -1258,7 +1298,7 @@ static const JanetReg os_cfuns[] = {
              "\t:blocks - number of blocks in file. 0 on windows\n"
              "\t:blocksize - size of blocks in file. 0 on windows\n"
              "\t:accessed - timestamp when file last accessed\n"
-             "\t:changed - timestamp when file last chnaged (permissions changed)\n"
+             "\t:changed - timestamp when file last changed (permissions changed)\n"
              "\t:modified - timestamp when file last modified (content changed)\n")
     },
     {
@@ -1270,9 +1310,9 @@ static const JanetReg os_cfuns[] = {
         "os/chmod", os_chmod,
         JDOC("(os/chmod path mode)\n\n"
              "Change file permissions, where mode is a permission string as returned by "
-             "os/stat, or an integer. "
-             "When mode is an integer, it is interpreted as a unix permission value, best specified in octal, like "
-             "8r666 or 8r400. Windows will not differentiate between user, group, and other permissions. Returns nil.")
+             "os/perm-string, or an integer as returned by os/perm-int. "
+             "When mode is an integer, it is interpreted as a Unix permission value, best specified in octal, like "
+             "8r666 or 8r400. Windows will not differentiate between user, group, and other permissions, and thus will combine all of these permissions. Returns nil.")
     },
     {
         "os/touch", os_touch,
@@ -1285,11 +1325,18 @@ static const JanetReg os_cfuns[] = {
         JDOC("(os/cd path)\n\n"
              "Change current directory to path. Returns nil on success, errors on failure.")
     },
+#ifndef JANET_NO_UMASK
+    {
+        "os/umask", os_umask,
+        JDOC("(os/umask mask)\n\n"
+             "Set a new umask, returns the old umask.")
+    },
+#endif
     {
         "os/mkdir", os_mkdir,
         JDOC("(os/mkdir path)\n\n"
              "Create a new directory. The path will be relative to the current directory if relative, otherwise "
-             "it will be an absolute path. Returns true if the directory was create, false if the directoyr already exists, and "
+             "it will be an absolute path. Returns true if the directory was created, false if the directory already exists, and "
              "errors otherwise.")
     },
     {
@@ -1305,9 +1352,12 @@ static const JanetReg os_cfuns[] = {
     {
         "os/link", os_link,
         JDOC("(os/link oldpath newpath &opt symlink)\n\n"
-             "Create a symlink from oldpath to newpath, returning nil. The 3rd optional paramater "
-             "enables a symlink iff truthy, hard link otherwise or if not provided. Does not work on Windows.")
+             "Create a link at newpath that points to oldpath and returns nil. "
+             "Iff symlink is truthy, creates a symlink. "
+             "Iff symlink is falsey or not provided, "
+             "creates a hard link. Does not work on Windows.")
     },
+#ifndef JANET_NO_SYMLINKS
     {
         "os/symlink", os_symlink,
         JDOC("(os/symlink oldpath newpath)\n\n"
@@ -1318,6 +1368,8 @@ static const JanetReg os_cfuns[] = {
         JDOC("(os/readlink path)\n\n"
              "Read the contents of a symbolic link. Does not work on Windows.\n")
     },
+#endif
+#ifndef JANET_NO_PROCESSES
     {
         "os/execute", os_execute,
         JDOC("(os/execute args &opts flags env)\n\n"
@@ -1335,6 +1387,7 @@ static const JanetReg os_cfuns[] = {
         JDOC("(os/shell str)\n\n"
              "Pass a command string str directly to the system shell.")
     },
+#endif
     {
         "os/setenv", os_setenv,
         JDOC("(os/setenv variable value)\n\n"
@@ -1376,14 +1429,14 @@ static const JanetReg os_cfuns[] = {
     {
         "os/cryptorand", os_cryptorand,
         JDOC("(os/cryptorand n &opt buf)\n\n"
-             "Get or append n bytes of good quality random data provided by the os. Returns a new buffer or buf.")
+             "Get or append n bytes of good quality random data provided by the OS. Returns a new buffer or buf.")
     },
     {
         "os/date", os_date,
         JDOC("(os/date &opt time local)\n\n"
              "Returns the given time as a date struct, or the current time if no time is given. "
              "Returns a struct with following key values. Note that all numbers are 0-indexed. "
-             "Date is given in UTC unless local is truthy, in which case the date is formated for "
+             "Date is given in UTC unless local is truthy, in which case the date is formatted for "
              "the local timezone.\n\n"
              "\t:seconds - number of seconds [0-61]\n"
              "\t:minutes - number of minutes [0-59]\n"
@@ -1406,6 +1459,19 @@ static const JanetReg os_cfuns[] = {
              "Get the absolute path for a given path, following ../, ./, and symlinks. "
              "Returns an absolute path as a string. Will raise an error on Windows.")
     },
+    {
+        "os/perm-string", os_permission_string,
+        JDOC("(os/perm-string int)\n\n"
+             "Convert a Unix octal permission value from a permission integer as returned by os/stat "
+             "to a human readable string, that follows the formatting "
+             "of unix tools like ls. Returns the string as a 9 character string of r, w, x and - characters. Does not "
+             "include the file/directory/symlink character as rendered by `ls`.")
+    },
+    {
+        "os/perm-int", os_permission_int,
+        JDOC("(os/perm-int bytes)\n\n"
+             "Parse a 9 character permission string and return an integer that can be used by chmod.")
+    },
 #endif
     {NULL, NULL, NULL}
 };

src/core/parse.c

@@ -26,6 +26,9 @@
 #include "util.h"
 #endif
 
+#define JANET_PARSER_DEAD 0x1
+#define JANET_PARSER_GENERATED_ERROR 0x2
+
 /* Check if a character is whitespace */
 static int is_whitespace(uint8_t c) {
     return c == ' '
@@ -164,12 +167,12 @@ static void popstate(JanetParser *p, Janet val) {
     for (;;) {
         JanetParseState top = p->states[--p->statecount];
         JanetParseState *newtop = p->states + p->statecount - 1;
+        /* Source mapping info */
+        if (janet_checktype(val, JANET_TUPLE)) {
+            janet_tuple_sm_line(janet_unwrap_tuple(val)) = (int32_t) top.line;
+            janet_tuple_sm_column(janet_unwrap_tuple(val)) = (int32_t) top.column;
+        }
         if (newtop->flags & PFLAG_CONTAINER) {
-            /* Source mapping info */
-            if (janet_checktype(val, JANET_TUPLE)) {
-                janet_tuple_sm_line(janet_unwrap_tuple(val)) = (int32_t) top.line;
-                janet_tuple_sm_column(janet_unwrap_tuple(val)) = (int32_t) top.column;
-            }
             newtop->argn++;
             /* Keep track of number of values in the root state */
             if (p->statecount == 1) p->pending++;
@@ -201,6 +204,8 @@ static int checkescape(uint8_t c) {
         default:
             return -1;
         case 'x':
+        case 'u':
+        case 'U':
             return 1;
         case 'n':
             return '\n';
@@ -228,6 +233,24 @@ static int checkescape(uint8_t c) {
 /* Forward declare */
 static int stringchar(JanetParser *p, JanetParseState *state, uint8_t c);
 
+static void write_codepoint(JanetParser *p, int32_t codepoint) {
+    if (codepoint <= 0x7F) {
+        push_buf(p, (uint8_t) codepoint);
+    } else if (codepoint <= 0x7FF) {
+        push_buf(p, (uint8_t)((codepoint >>  6) & 0x1F) | 0xC0);
+        push_buf(p, (uint8_t)((codepoint >>  0) & 0x3F) | 0x80);
+    } else if (codepoint <= 0xFFFF) {
+        push_buf(p, (uint8_t)((codepoint >> 12) & 0x0F) | 0xE0);
+        push_buf(p, (uint8_t)((codepoint >>  6) & 0x3F) | 0x80);
+        push_buf(p, (uint8_t)((codepoint >>  0) & 0x3F) | 0x80);
+    } else {
+        push_buf(p, (uint8_t)((codepoint >> 18) & 0x07) | 0xF0);
+        push_buf(p, (uint8_t)((codepoint >> 12) & 0x3F) | 0x80);
+        push_buf(p, (uint8_t)((codepoint >>  6) & 0x3F) | 0x80);
+        push_buf(p, (uint8_t)((codepoint >>  0) & 0x3F) | 0x80);
+    }
+}
+
 static int escapeh(JanetParser *p, JanetParseState *state, uint8_t c) {
     int digit = to_hex(c);
     if (digit < 0) {
@@ -237,7 +260,27 @@ static int escapeh(JanetParser *p, JanetParseState *state, uint8_t c) {
     state->argn = (state->argn << 4) + digit;
     state->counter--;
     if (!state->counter) {
-        push_buf(p, (state->argn & 0xFF));
+        push_buf(p, (uint8_t)(state->argn & 0xFF));
+        state->argn = 0;
+        state->consumer = stringchar;
+    }
+    return 1;
+}
+
+static int escapeu(JanetParser *p, JanetParseState *state, uint8_t c) {
+    int digit = to_hex(c);
+    if (digit < 0) {
+        p->error = "invalid hex digit in unicode escape";
+        return 1;
+    }
+    state->argn = (state->argn << 4) + digit;
+    state->counter--;
+    if (!state->counter) {
+        if (state->argn > 0x10FFFF) {
+            p->error = "invalid unicode codepoint";
+            return 1;
+        }
+        write_codepoint(p, state->argn);
         state->argn = 0;
         state->consumer = stringchar;
     }
@@ -254,6 +297,10 @@ static int escape1(JanetParser *p, JanetParseState *state, uint8_t c) {
         state->counter = 2;
         state->argn = 0;
         state->consumer = escapeh;
+    } else if (c == 'u' || c == 'U') {
+        state->counter = c == 'u' ? 4 : 6;
+        state->argn = 0;
+        state->consumer = escapeu;
     } else {
         push_buf(p, (uint8_t) e);
         state->consumer = stringchar;
@@ -393,21 +440,23 @@ static Janet close_array(JanetParser *p, JanetParseState *state) {
 
 static Janet close_struct(JanetParser *p, JanetParseState *state) {
     JanetKV *st = janet_struct_begin(state->argn >> 1);
-    for (int32_t i = state->argn; i > 0; i -= 2) {
-        Janet value = p->args[--p->argcount];
-        Janet key = p->args[--p->argcount];
+    for (size_t i = p->argcount - state->argn; i < p->argcount; i += 2) {
+        Janet key = p->args[i];
+        Janet value = p->args[i + 1];
         janet_struct_put(st, key, value);
     }
+    p->argcount -= state->argn;
     return janet_wrap_struct(janet_struct_end(st));
 }
 
 static Janet close_table(JanetParser *p, JanetParseState *state) {
     JanetTable *table = janet_table(state->argn >> 1);
-    for (int32_t i = state->argn; i > 0; i -= 2) {
-        Janet value = p->args[--p->argcount];
-        Janet key = p->args[--p->argcount];
+    for (size_t i = p->argcount - state->argn; i < p->argcount; i += 2) {
+        Janet key = p->args[i];
+        Janet value = p->args[i + 1];
         janet_table_put(table, key, value);
     }
+    p->argcount -= state->argn;
     return janet_wrap_table(table);
 }
 
@@ -591,11 +640,30 @@ void janet_parser_eof(JanetParser *parser) {
     size_t oldline = parser->line;
     janet_parser_consume(parser, '\n');
     if (parser->statecount > 1) {
-        parser->error = "unexpected end of source";
+        JanetParseState *s = parser->states + (parser->statecount - 1);
+        JanetBuffer *buffer = janet_buffer(40);
+        janet_buffer_push_cstring(buffer, "unexpected end of source, ");
+        if (s->flags & PFLAG_PARENS) {
+            janet_buffer_push_u8(buffer, '(');
+        } else if (s->flags & PFLAG_SQRBRACKETS) {
+            janet_buffer_push_u8(buffer, '[');
+        } else if (s->flags & PFLAG_CURLYBRACKETS) {
+            janet_buffer_push_u8(buffer, '{');
+        } else if (s->flags & PFLAG_STRING) {
+            janet_buffer_push_u8(buffer, '"');
+        } else if (s->flags & PFLAG_LONGSTRING) {
+            int32_t i;
+            for (i = 0; i < s->argn; i++) {
+                janet_buffer_push_u8(buffer, '`');
+            }
+        }
+        janet_formatb(buffer, " opened at line %d, column %d", s->line, s->column);
+        parser->error = (const char *) janet_string(buffer->data, buffer->count);
+        parser->flag |= JANET_PARSER_GENERATED_ERROR;
     }
     parser->line = oldline;
     parser->column = oldcolumn;
-    parser->flag = 1;
+    parser->flag |= JANET_PARSER_DEAD;
 }
 
 enum JanetParserStatus janet_parser_status(JanetParser *parser) {
@@ -617,6 +685,7 @@ const char *janet_parser_error(JanetParser *parser) {
     if (status == JANET_PARSE_ERROR) {
         const char *e = parser->error;
         parser->error = NULL;
+        parser->flag &= ~JANET_PARSER_GENERATED_ERROR;
         janet_parser_flush(parser);
         return e;
     }
@@ -720,6 +789,9 @@ static int parsermark(void *p, size_t size) {
     for (i = 0; i < parser->argcount; i++) {
         janet_mark(parser->args[i]);
     }
+    if (parser->flag & JANET_PARSER_GENERATED_ERROR) {
+        janet_mark(janet_wrap_string((const uint8_t *) parser->error));
+    }
     return 0;
 }
 
@@ -854,7 +926,11 @@ static Janet cfun_parse_error(int32_t argc, Janet *argv) {
     janet_fixarity(argc, 1);
     JanetParser *p = janet_getabstract(argv, 0, &janet_parser_type);
     const char *err = janet_parser_error(p);
-    if (err) return janet_cstringv(err);
+    if (err) {
+        return (p->flag & JANET_PARSER_GENERATED_ERROR)
+               ? janet_wrap_string((const uint8_t *) err)
+               : janet_cstringv(err);
+    }
     return janet_wrap_nil();
 }
 
@@ -953,31 +1029,30 @@ struct ParserStateGetter {
 };
 
 static Janet parser_state_delimiters(const JanetParser *_p) {
-    JanetParser *clone = janet_abstract(&janet_parser_type, sizeof(JanetParser));
-    janet_parser_clone(_p, clone);
+    JanetParser *p = (JanetParser *)_p;
     size_t i;
     const uint8_t *str;
     size_t oldcount;
-    oldcount = clone->bufcount;
-    for (i = 0; i < clone->statecount; i++) {
-        JanetParseState *s = clone->states + i;
+    oldcount = p->bufcount;
+    for (i = 0; i < p->statecount; i++) {
+        JanetParseState *s = p->states + i;
         if (s->flags & PFLAG_PARENS) {
-            push_buf(clone, '(');
+            push_buf(p, '(');
         } else if (s->flags & PFLAG_SQRBRACKETS) {
-            push_buf(clone, '[');
+            push_buf(p, '[');
         } else if (s->flags & PFLAG_CURLYBRACKETS) {
-            push_buf(clone, '{');
+            push_buf(p, '{');
         } else if (s->flags & PFLAG_STRING) {
-            push_buf(clone, '"');
+            push_buf(p, '"');
         } else if (s->flags & PFLAG_LONGSTRING) {
             int32_t i;
             for (i = 0; i < s->argn; i++) {
-                push_buf(clone, '`');
+                push_buf(p, '`');
             }
         }
     }
-    str = janet_string(clone->buf + oldcount, (int32_t)(clone->bufcount - oldcount));
-    clone->bufcount = oldcount;
+    str = janet_string(p->buf + oldcount, (int32_t)(p->bufcount - oldcount));
+    p->bufcount = oldcount;
     return janet_wrap_string(str);
 }
 
@@ -1064,7 +1139,7 @@ static const JanetReg parse_cfuns[] = {
         "parser/new", cfun_parse_parser,
         JDOC("(parser/new)\n\n"
              "Creates and returns a new parser object. Parsers are state machines "
-             "that can receive bytes, and generate a stream of janet values.")
+             "that can receive bytes, and generate a stream of values.")
     },
     {
         "parser/clone", cfun_parse_clone,

src/core/peg.c

@@ -150,6 +150,7 @@ tail:
             down1(s);
             const uint8_t *result = peg_rule(s, s->bytecode + rule[2], text);
             up1(s);
+            text -= ((int32_t *)rule)[1];
             return result ? text : NULL;
         }
 
@@ -205,6 +206,29 @@ tail:
             return (result) ? NULL : text;
         }
 
+        case RULE_THRU:
+        case RULE_TO: {
+            const uint32_t *rule_a = s->bytecode + rule[1];
+            const uint8_t *next_text;
+            CapState cs = cap_save(s);
+            down1(s);
+            while (text < s->text_end) {
+                CapState cs2 = cap_save(s);
+                next_text = peg_rule(s, rule_a, text);
+                if (next_text) {
+                    if (rule[0] == RULE_TO) cap_load(s, cs2);
+                    break;
+                }
+                text++;
+            }
+            up1(s);
+            if (text >= s->text_end) {
+                cap_load(s, cs);
+                return NULL;
+            }
+            return rule[0] == RULE_TO ? text : next_text;
+        }
+
         case RULE_BETWEEN: {
             uint32_t lo = rule[1];
             uint32_t hi = rule[2];
@@ -413,6 +437,38 @@ tail:
             return NULL;
         }
 
+        case RULE_LENPREFIX: {
+            int oldmode = s->mode;
+            s->mode = PEG_MODE_NORMAL;
+            const uint8_t *next_text;
+            CapState cs = cap_save(s);
+            down1(s);
+            next_text = peg_rule(s, s->bytecode + rule[1], text);
+            up1(s);
+            if (NULL == next_text) return NULL;
+            s->mode = oldmode;
+            int32_t num_sub_captures = s->captures->count - cs.cap;
+            Janet lencap;
+            if (num_sub_captures <= 0 ||
+                    (lencap = s->captures->data[cs.cap], !janet_checkint(lencap))) {
+                cap_load(s, cs);
+                return NULL;
+            }
+            int32_t nrep = janet_unwrap_integer(lencap);
+            /* drop captures from len pattern */
+            cap_load(s, cs);
+            for (int32_t i = 0; i < nrep; i++) {
+                down1(s);
+                next_text = peg_rule(s, s->bytecode + rule[2], next_text);
+                up1(s);
+                if (NULL == next_text) {
+                    cap_load(s, cs);
+                    return NULL;
+                }
+            }
+            return next_text;
+        }
+
     }
 }
 
@@ -657,6 +713,9 @@ static void spec_if(Builder *b, int32_t argc, const Janet *argv) {
 static void spec_ifnot(Builder *b, int32_t argc, const Janet *argv) {
     spec_branch(b, argc, argv, RULE_IFNOT);
 }
+static void spec_lenprefix(Builder *b, int32_t argc, const Janet *argv) {
+    spec_branch(b, argc, argv, RULE_LENPREFIX);
+}
 
 static void spec_between(Builder *b, int32_t argc, const Janet *argv) {
     peg_fixarity(b, argc, 3);
@@ -729,6 +788,12 @@ static void spec_error(Builder *b, int32_t argc, const Janet *argv) {
 static void spec_drop(Builder *b, int32_t argc, const Janet *argv) {
     spec_onerule(b, argc, argv, RULE_DROP);
 }
+static void spec_to(Builder *b, int32_t argc, const Janet *argv) {
+    spec_onerule(b, argc, argv, RULE_TO);
+}
+static void spec_thru(Builder *b, int32_t argc, const Janet *argv) {
+    spec_onerule(b, argc, argv, RULE_THRU);
+}
 
 /* Rule of the form [rule, tag] */
 static void spec_cap1(Builder *b, int32_t argc, const Janet *argv, uint32_t op) {
@@ -847,6 +912,7 @@ static const SpecialPair peg_specials[] = {
     {"group", spec_group},
     {"if", spec_if},
     {"if-not", spec_ifnot},
+    {"lenprefix", spec_lenprefix},
     {"look", spec_look},
     {"not", spec_not},
     {"opt", spec_opt},
@@ -858,6 +924,8 @@ static const SpecialPair peg_specials[] = {
     {"sequence", spec_sequence},
     {"set", spec_set},
     {"some", spec_some},
+    {"thru", spec_thru},
+    {"to", spec_to},
 };
 
 /* Compile a janet value into a rule and return the rule index. */
@@ -960,6 +1028,14 @@ static uint32_t peg_compile1(Builder *b, Janet peg) {
             const Janet *tup = janet_unwrap_tuple(peg);
             int32_t len = janet_tuple_length(tup);
             if (len == 0) peg_panic(b, "tuple in grammar must have non-zero length");
+            if (janet_checkint(tup[0])) {
+                int32_t n = janet_unwrap_integer(tup[0]);
+                if (n < 0) {
+                    peg_panicf(b, "expected non-negative integer, got %d", n);
+                }
+                spec_repeat(b, len, tup);
+                break;
+            }
             if (!janet_checktype(tup[0], JANET_SYMBOL))
                 peg_panicf(b, "expected grammar command, found %v", tup[0]);
             const uint8_t *sym = janet_unwrap_symbol(tup[0]);
@@ -1100,6 +1176,7 @@ static void *peg_unmarshal(JanetMarshalContext *ctx) {
             break;
             case RULE_IF:
             case RULE_IFNOT:
+            case RULE_LENPREFIX:
                 /* [rule_a, rule_b (b if not a)] */
                 if (rule[1] >= blen) goto bad;
                 if (rule[2] >= blen) goto bad;
@@ -1142,6 +1219,8 @@ static void *peg_unmarshal(JanetMarshalContext *ctx) {
             case RULE_ERROR:
             case RULE_DROP:
             case RULE_NOT:
+            case RULE_TO:
+            case RULE_THRU:
                 /* [rule] */
                 if (rule[1] >= blen) goto bad;
                 op_flags[rule[1]] |= 0x01;
@@ -1229,47 +1308,136 @@ static Janet cfun_peg_compile(int32_t argc, Janet *argv) {
     return janet_wrap_abstract(peg);
 }
 
-static Janet cfun_peg_match(int32_t argc, Janet *argv) {
-    janet_arity(argc, 2, -1);
+/* Common data for peg cfunctions */
+typedef struct {
     JanetPeg *peg;
+    PegState s;
+    JanetByteView bytes;
+    JanetByteView repl;
+    int32_t start;
+} PegCall;
+
+/* Initialize state for peg cfunctions */
+static PegCall peg_cfun_init(int32_t argc, Janet *argv, int get_replace) {
+    PegCall ret;
+    int32_t min = get_replace ? 3 : 2;
+    janet_arity(argc, get_replace, -1);
     if (janet_checktype(argv[0], JANET_ABSTRACT) &&
             janet_abstract_type(janet_unwrap_abstract(argv[0])) == &janet_peg_type) {
-        peg = janet_unwrap_abstract(argv[0]);
+        ret.peg = janet_unwrap_abstract(argv[0]);
     } else {
-        peg = compile_peg(argv[0]);
+        ret.peg = compile_peg(argv[0]);
     }
-    JanetByteView bytes = janet_getbytes(argv, 1);
-    int32_t start;
-    PegState s;
-    if (argc > 2) {
-        start = janet_gethalfrange(argv, 2, bytes.len, "offset");
-        s.extrac = argc - 3;
-        s.extrav = janet_tuple_n(argv + 3, argc - 3);
+    if (get_replace) {
+        ret.repl = janet_getbytes(argv, 1);
+        ret.bytes = janet_getbytes(argv, 2);
     } else {
-        start = 0;
-        s.extrac = 0;
-        s.extrav = NULL;
+        ret.bytes = janet_getbytes(argv, 1);
     }
-    s.mode = PEG_MODE_NORMAL;
-    s.text_start = bytes.bytes;
-    s.text_end = bytes.bytes + bytes.len;
-    s.depth = JANET_RECURSION_GUARD;
-    s.captures = janet_array(0);
-    s.scratch = janet_buffer(10);
-    s.tags = janet_buffer(10);
-    s.constants = peg->constants;
-    s.bytecode = peg->bytecode;
-    const uint8_t *result = peg_rule(&s, s.bytecode, bytes.bytes + start);
-    return result ? janet_wrap_array(s.captures) : janet_wrap_nil();
+    if (argc > min) {
+        ret.start = janet_gethalfrange(argv, min, ret.bytes.len, "offset");
+        ret.s.extrac = argc - min - 1;
+        ret.s.extrav = janet_tuple_n(argv + min + 1, argc - min - 1);
+    } else {
+        ret.start = 0;
+        ret.s.extrac = 0;
+        ret.s.extrav = NULL;
+    }
+    ret.s.mode = PEG_MODE_NORMAL;
+    ret.s.text_start = ret.bytes.bytes;
+    ret.s.text_end = ret.bytes.bytes + ret.bytes.len;
+    ret.s.depth = JANET_RECURSION_GUARD;
+    ret.s.captures = janet_array(0);
+    ret.s.scratch = janet_buffer(10);
+    ret.s.tags = janet_buffer(10);
+    ret.s.constants = ret.peg->constants;
+    ret.s.bytecode = ret.peg->bytecode;
+    return ret;
 }
 
+static void peg_call_reset(PegCall *c) {
+    c->s.captures->count = 0;
+    c->s.scratch->count = 0;
+    c->s.tags->count = 0;
+}
+
+static Janet cfun_peg_match(int32_t argc, Janet *argv) {
+    PegCall c = peg_cfun_init(argc, argv, 0);
+    const uint8_t *result = peg_rule(&c.s, c.s.bytecode, c.bytes.bytes + c.start);
+    return result ? janet_wrap_array(c.s.captures) : janet_wrap_nil();
+}
+
+static Janet cfun_peg_find(int32_t argc, Janet *argv) {
+    PegCall c = peg_cfun_init(argc, argv, 0);
+    for (int32_t i = c.start; i < c.bytes.len; i++) {
+        peg_call_reset(&c);
+        if (peg_rule(&c.s, c.s.bytecode, c.bytes.bytes + i))
+            return janet_wrap_integer(i);
+    }
+    return janet_wrap_nil();
+}
+
+static Janet cfun_peg_find_all(int32_t argc, Janet *argv) {
+    PegCall c = peg_cfun_init(argc, argv, 0);
+    JanetArray *ret = janet_array(0);
+    for (int32_t i = c.start; i < c.bytes.len; i++) {
+        peg_call_reset(&c);
+        if (peg_rule(&c.s, c.s.bytecode, c.bytes.bytes + i))
+            janet_array_push(ret, janet_wrap_integer(i));
+    }
+    return janet_wrap_array(ret);
+}
+
+static Janet cfun_peg_replace_generic(int32_t argc, Janet *argv, int only_one) {
+    PegCall c = peg_cfun_init(argc, argv, 1);
+    JanetBuffer *ret = janet_buffer(0);
+    int32_t trail = 0;
+    for (int32_t i = c.start; i < c.bytes.len;) {
+        peg_call_reset(&c);
+        const uint8_t *result = peg_rule(&c.s, c.s.bytecode, c.bytes.bytes + i);
+        if (NULL != result) {
+            if (trail < i) {
+                janet_buffer_push_bytes(ret, c.bytes.bytes + trail, (i - trail));
+                trail = i;
+            }
+            int32_t nexti = (int32_t)(result - c.bytes.bytes);
+            janet_buffer_push_bytes(ret, c.repl.bytes, c.repl.len);
+            trail = nexti;
+            if (nexti == i) nexti++;
+            i = nexti;
+            if (only_one) break;
+        } else {
+            i++;
+        }
+    }
+    if (trail < c.bytes.len) {
+        janet_buffer_push_bytes(ret, c.bytes.bytes + trail, (c.bytes.len - trail));
+    }
+    return janet_wrap_buffer(ret);
+}
+
+static Janet cfun_peg_replace_all(int32_t argc, Janet *argv) {
+    return cfun_peg_replace_generic(argc, argv, 0);
+}
+
+static Janet cfun_peg_replace(int32_t argc, Janet *argv) {
+    return cfun_peg_replace_generic(argc, argv, 1);
+}
+
+static JanetMethod peg_methods[] = {
+    {"match", cfun_peg_match},
+    {"find", cfun_peg_find},
+    {"find-all", cfun_peg_find_all},
+    {"replace", cfun_peg_replace},
+    {"replace-all", cfun_peg_replace_all},
+    {NULL, NULL}
+};
+
 static int cfun_peg_getter(JanetAbstract a, Janet key, Janet *out) {
     (void) a;
-    if (janet_keyeq(key, "match")) {
-        *out = janet_wrap_cfunction(cfun_peg_match);
-        return 1;
-    }
-    return 0;
+    if (!janet_checktype(key, JANET_KEYWORD))
+        return 0;
+    return janet_getmethod(janet_unwrap_keyword(key), peg_methods, out);
 }
 
 static const JanetReg peg_cfuns[] = {
@@ -1283,8 +1451,28 @@ static const JanetReg peg_cfuns[] = {
         "peg/match", cfun_peg_match,
         JDOC("(peg/match peg text &opt start & args)\n\n"
              "Match a Parsing Expression Grammar to a byte string and return an array of captured values. "
-             "Returns nil if text does not match the language defined by peg. The syntax of PEGs are very "
-             "similar to those defined by LPeg, and have similar capabilities.")
+             "Returns nil if text does not match the language defined by peg. The syntax of PEGs is documented on the Janet website.")
+    },
+    {
+        "peg/find", cfun_peg_find,
+        JDOC("(peg/find peg text &opt start & args)\n\n"
+             "Find first index where the peg matches in text. Returns an integer, or nil if not found.")
+    },
+    {
+        "peg/find-all", cfun_peg_find_all,
+        JDOC("(peg/find-all peg text &opt start & args)\n\n"
+             "Find all indexes where the peg matches in text. Returns an array of integers.")
+    },
+    {
+        "peg/replace", cfun_peg_replace,
+        JDOC("(peg/replace peg repl text &opt start & args)\n\n"
+             "Replace first match of peg in text with repl, returning a new buffer. The peg does not need to make captures to do replacement. "
+             "If no matches are found, returns the input string in a new buffer.")
+    },
+    {
+        "peg/replace-all", cfun_peg_replace_all,
+        JDOC("(peg/replace-all peg repl text &opt start & args)\n\n"
+             "Replace all matches of peg in text with repl, returning a new buffer. The peg does not need to make captures to do replacement.")
     },
     {NULL, NULL, NULL}
 };

src/core/pp.c

@@ -39,11 +39,9 @@
 
 static void number_to_string_b(JanetBuffer *buffer, double x) {
     janet_buffer_ensure(buffer, buffer->count + BUFSIZE, 2);
-    /* Use int32_t range for valid integers because that is the
-     * range most integer-expecting functions in the C api use. */
     const char *fmt = (x == floor(x) &&
-                       x <= ((double) INT32_MAX) &&
-                       x >= ((double) INT32_MIN)) ? "%.0f" : "%g";
+                       x <= JANET_INTMAX_DOUBLE &&
+                       x >= JANET_INTMIN_DOUBLE) ? "%.0f" : "%g";
     int count = snprintf((char *) buffer->data + buffer->count, BUFSIZE, fmt, x);
     buffer->count += count;
 }
@@ -123,9 +121,6 @@ static void string_description_b(JanetBuffer *buffer, const char *title, void *p
 #undef POINTSIZE
 }
 
-#undef HEX
-#undef BUFSIZE
-
 static void janet_escape_string_impl(JanetBuffer *buffer, const uint8_t *str, int32_t len) {
     janet_buffer_push_u8(buffer, '"');
     for (int32_t i = 0; i < len; ++i) {
@@ -155,8 +150,11 @@ static void janet_escape_string_impl(JanetBuffer *buffer, const uint8_t *str, in
             case '\\':
                 janet_buffer_push_bytes(buffer, (const uint8_t *)"\\\\", 2);
                 break;
+            case '\t':
+                janet_buffer_push_bytes(buffer, (const uint8_t *)"\\t", 2);
+                break;
             default:
-                if (c < 32 || c > 127) {
+                if (c < 32 || c > 126) {
                     uint8_t buf[4];
                     buf[0] = '\\';
                     buf[1] = 'x';
@@ -188,7 +186,7 @@ static void janet_escape_buffer_b(JanetBuffer *buffer, JanetBuffer *bx) {
 void janet_to_string_b(JanetBuffer *buffer, Janet x) {
     switch (janet_type(x)) {
         case JANET_NIL:
-            janet_buffer_push_cstring(buffer, "nil");
+            janet_buffer_push_cstring(buffer, "");
             break;
         case JANET_BOOLEAN:
             janet_buffer_push_cstring(buffer,
@@ -277,6 +275,9 @@ void janet_description_b(JanetBuffer *buffer, Janet x) {
     switch (janet_type(x)) {
         default:
             break;
+        case JANET_NIL:
+            janet_buffer_push_cstring(buffer, "nil");
+            return;
         case JANET_KEYWORD:
             janet_buffer_push_u8(buffer, ':');
             break;
@@ -351,12 +352,16 @@ static int print_jdn_one(struct pretty *S, Janet x, int depth) {
     if (depth == 0) return 1;
     switch (janet_type(x)) {
         case JANET_NIL:
-        case JANET_NUMBER:
         case JANET_BOOLEAN:
         case JANET_BUFFER:
         case JANET_STRING:
             janet_description_b(S->buffer, x);
             break;
+        case JANET_NUMBER:
+            janet_buffer_ensure(S->buffer, S->buffer->count + BUFSIZE, 2);
+            int count = snprintf((char *) S->buffer->data + S->buffer->count, BUFSIZE, "%.17g", janet_unwrap_number(x));
+            S->buffer->count += count;
+            break;
         case JANET_SYMBOL:
         case JANET_KEYWORD:
             if (contains_bad_chars(janet_unwrap_keyword(x), janet_type(x) == JANET_SYMBOL)) return 1;
@@ -459,8 +464,8 @@ static const char *janet_pretty_colors[] = {
 
 #define JANET_PRETTY_DICT_ONELINE 4
 #define JANET_PRETTY_IND_ONELINE 10
-#define JANET_PRETTY_DICT_LIMIT 16
-#define JANET_PRETTY_ARRAY_LIMIT 16
+#define JANET_PRETTY_DICT_LIMIT 30
+#define JANET_PRETTY_ARRAY_LIMIT 160
 
 /* Helper for pretty printing */
 static void janet_pretty_one(struct pretty *S, Janet x, int is_dict_value) {
@@ -527,7 +532,7 @@ static void janet_pretty_one(struct pretty *S, Janet x, int is_dict_value) {
                 if (!isarray && !(S->flags & JANET_PRETTY_ONELINE) && len >= JANET_PRETTY_IND_ONELINE)
                     janet_buffer_push_u8(S->buffer, ' ');
                 if (is_dict_value && len >= JANET_PRETTY_IND_ONELINE) print_newline(S, 0);
-                if (len > JANET_PRETTY_ARRAY_LIMIT) {
+                if (len > JANET_PRETTY_ARRAY_LIMIT && !(S->flags & JANET_PRETTY_NOTRUNC)) {
                     for (i = 0; i < 3; i++) {
                         if (i) print_newline(S, 0);
                         janet_pretty_one(S, arr[i], 0);
@@ -591,6 +596,11 @@ static void janet_pretty_one(struct pretty *S, Janet x, int is_dict_value) {
                 if (is_dict_value && len >= JANET_PRETTY_DICT_ONELINE) print_newline(S, 0);
                 for (i = 0; i < cap; i++) {
                     if (!janet_checktype(kvs[i].key, JANET_NIL)) {
+                        if (counter == JANET_PRETTY_DICT_LIMIT && !(S->flags & JANET_PRETTY_NOTRUNC)) {
+                            print_newline(S, 0);
+                            janet_buffer_push_cstring(S->buffer, "...");
+                            break;
+                        }
                         if (first_kv_pair) {
                             first_kv_pair = 0;
                         } else {
@@ -600,11 +610,6 @@ static void janet_pretty_one(struct pretty *S, Janet x, int is_dict_value) {
                         janet_buffer_push_u8(S->buffer, ' ');
                         janet_pretty_one(S, kvs[i].value, 1);
                         counter++;
-                        if (counter == 10) {
-                            print_newline(S, 0);
-                            janet_buffer_push_cstring(S->buffer, "...");
-                            break;
-                        }
                     }
                 }
             }
@@ -728,7 +733,7 @@ static const char *scanformat(
     return p;
 }
 
-void janet_formatb(JanetBuffer *b, const char *format, va_list args) {
+void janet_formatbv(JanetBuffer *b, const char *format, va_list args) {
     const char *format_end = format + strlen(format);
     const char *c = format;
     int32_t startlen = b->count;
@@ -751,7 +756,6 @@ void janet_formatb(JanetBuffer *b, const char *format, va_list args) {
                 case 'd':
                 case 'i':
                 case 'o':
-                case 'u':
                 case 'x':
                 case 'X': {
                     int32_t n = va_arg(args, long);
@@ -802,6 +806,10 @@ void janet_formatb(JanetBuffer *b, const char *format, va_list args) {
                     pushtypes(b, types);
                     break;
                 }
+                case 'M':
+                case 'm':
+                case 'N':
+                case 'n':
                 case 'Q':
                 case 'q':
                 case 'P':
@@ -809,11 +817,13 @@ void janet_formatb(JanetBuffer *b, const char *format, va_list args) {
                     int depth = atoi(precision);
                     if (depth < 1) depth = 4;
                     char d = c[-1];
-                    int has_color = (d == 'P') || (d == 'Q');
-                    int has_oneline = (d == 'Q') || (d == 'q');
+                    int has_color = (d == 'P') || (d == 'Q') || (d == 'M') || (d == 'N');
+                    int has_oneline = (d == 'Q') || (d == 'q') || (d == 'N') || (d == 'n');
+                    int has_notrunc = (d == 'M') || (d == 'm') || (d == 'N') || (d == 'n');
                     int flags = 0;
                     flags |= has_color ? JANET_PRETTY_COLOR : 0;
                     flags |= has_oneline ? JANET_PRETTY_ONELINE : 0;
+                    flags |= has_notrunc ? JANET_PRETTY_NOTRUNC : 0;
                     janet_pretty_(b, depth, flags, va_arg(args, Janet), startlen);
                     break;
                 }
@@ -853,7 +863,7 @@ const uint8_t *janet_formatc(const char *format, ...) {
     va_start(args, format);
 
     /* Run format */
-    janet_formatb(&buffer, format, args);
+    janet_formatbv(&buffer, format, args);
 
     /* Iterate length */
     va_end(args);
@@ -863,6 +873,14 @@ const uint8_t *janet_formatc(const char *format, ...) {
     return ret;
 }
 
+JanetBuffer *janet_formatb(JanetBuffer *buffer, const char *format, ...) {
+    va_list args;
+    va_start(args, format);
+    janet_formatbv(buffer, format, args);
+    va_end(args);
+    return buffer;
+}
+
 /* Shared implementation between string/format and
  * buffer/format */
 void janet_buffer_format(
@@ -896,7 +914,6 @@ void janet_buffer_format(
                 case 'd':
                 case 'i':
                 case 'o':
-                case 'u':
                 case 'x':
                 case 'X': {
                     int32_t n = janet_getinteger(argv, arg);
@@ -938,19 +955,24 @@ void janet_buffer_format(
                     janet_description_b(b, argv[arg]);
                     break;
                 }
+                case 'M':
+                case 'm':
+                case 'N':
+                case 'n':
                 case 'Q':
                 case 'q':
                 case 'P':
                 case 'p': { /* janet pretty , precision = depth */
                     int depth = atoi(precision);
-                    if (depth < 1)
-                        depth = 4;
-                    char c = strfrmt[-1];
-                    int has_color = (c == 'P') || (c == 'Q');
-                    int has_oneline = (c == 'Q') || (c == 'q');
+                    if (depth < 1) depth = 4;
+                    char d = strfrmt[-1];
+                    int has_color = (d == 'P') || (d == 'Q') || (d == 'M') || (d == 'N');
+                    int has_oneline = (d == 'Q') || (d == 'q') || (d == 'N') || (d == 'n');
+                    int has_notrunc = (d == 'M') || (d == 'm') || (d == 'N') || (d == 'n');
                     int flags = 0;
                     flags |= has_color ? JANET_PRETTY_COLOR : 0;
                     flags |= has_oneline ? JANET_PRETTY_ONELINE : 0;
+                    flags |= has_notrunc ? JANET_PRETTY_NOTRUNC : 0;
                     janet_pretty_(b, depth, flags, argv[arg], startlen);
                     break;
                 }
@@ -974,3 +996,6 @@ void janet_buffer_format(
         }
     }
 }
+
+#undef HEX
+#undef BUFSIZE

src/core/regalloc.c

@@ -145,7 +145,7 @@ void janetc_regalloc_free(JanetcRegisterAllocator *ra, int32_t reg) {
 int32_t janetc_regalloc_temp(JanetcRegisterAllocator *ra, JanetcRegisterTemp nth) {
     int32_t oldmax = ra->max;
     if (ra->regtemps & (1 << nth)) {
-        janet_exit("regtemp already allocated");
+        JANET_EXIT("regtemp already allocated");
     }
     ra->regtemps |= 1 << nth;
     int32_t reg = janetc_regalloc_1(ra);

src/core/run.c

@@ -23,7 +23,6 @@
 #ifndef JANET_AMALG
 #include "features.h"
 #include <janet.h>
-#include "state.h"
 #endif
 
 /* Run a string */
@@ -50,15 +49,16 @@ int janet_dobytes(JanetTable *env, const uint8_t *bytes, int32_t len, const char
                 JanetFiber *fiber = janet_fiber(f, 64, 0, NULL);
                 fiber->env = env;
                 JanetSignal status = janet_continue(fiber, janet_wrap_nil(), &ret);
-                if (status != JANET_SIGNAL_OK) {
+                if (status != JANET_SIGNAL_OK && status != JANET_SIGNAL_EVENT) {
                     janet_stacktrace(fiber, ret);
                     errflags |= 0x01;
                     done = 1;
                 }
             } else {
+                ret = janet_wrap_string(cres.error);
                 if (cres.macrofiber) {
                     janet_eprintf("compile error in %s: ", sourcePath);
-                    janet_stacktrace(cres.macrofiber, janet_wrap_string(cres.error));
+                    janet_stacktrace(cres.macrofiber, ret);
                 } else {
                     janet_eprintf("compile error in %s: %s\n", sourcePath,
                                   (const char *)cres.error);
@@ -68,25 +68,23 @@ int janet_dobytes(JanetTable *env, const uint8_t *bytes, int32_t len, const char
             }
         }
 
+        if (done) break;
+
         /* Dispatch based on parse state */
         switch (janet_parser_status(&parser)) {
             case JANET_PARSE_DEAD:
                 done = 1;
                 break;
-            case JANET_PARSE_ERROR:
+            case JANET_PARSE_ERROR: {
+                const char *e = janet_parser_error(&parser);
                 errflags |= 0x04;
-                janet_eprintf("parse error in %s: %s\n",
-                              sourcePath, janet_parser_error(&parser));
+                ret = janet_cstringv(e);
+                janet_eprintf("parse error in %s: %s\n", sourcePath, e);
                 done = 1;
                 break;
-            case JANET_PARSE_PENDING:
-                if (index == len) {
-                    janet_parser_eof(&parser);
-                } else {
-                    janet_parser_consume(&parser, bytes[index++]);
-                }
-                break;
+            }
             case JANET_PARSE_ROOT:
+            case JANET_PARSE_PENDING:
                 if (index >= len) {
                     janet_parser_eof(&parser);
                 } else {

src/core/specials.c

@@ -622,10 +622,11 @@ static JanetSlot janetc_while(JanetFopts opts, int32_t argn, const Janet *argv)
     /* Check if closure created in while scope. If so,
      * recompile in a function scope. */
     if (tempscope.flags & JANET_SCOPE_CLOSURE) {
+        subopts = janetc_fopts_default(c);
         tempscope.flags |= JANET_SCOPE_UNUSED;
         janetc_popscope(c);
-        janet_v__cnt(c->buffer) = labelwt;
-        janet_v__cnt(c->mapbuffer) = labelwt;
+        if (c->buffer) janet_v__cnt(c->buffer) = labelwt;
+        if (c->mapbuffer) janet_v__cnt(c->mapbuffer) = labelwt;
 
         janetc_scope(&tempscope, c, JANET_SCOPE_FUNCTION, "while-iife");
 
@@ -648,6 +649,7 @@ static JanetSlot janetc_while(JanetFopts opts, int32_t argn, const Janet *argv)
         /* Compile function */
         JanetFuncDef *def = janetc_pop_funcdef(c);
         def->name = janet_cstring("_while");
+        janet_def_addflags(def);
         int32_t defindex = janetc_addfuncdef(c, def);
         /* And then load the closure and call it. */
         int32_t cloreg = janetc_regalloc_temp(&c->scope->ra, JANETC_REGTEMP_0);
@@ -822,6 +824,7 @@ static JanetSlot janetc_fn(JanetFopts opts, int32_t argn, const Janet *argv) {
     if (structarg) def->flags |= JANET_FUNCDEF_FLAG_STRUCTARG;
 
     if (selfref) def->name = janet_unwrap_symbol(head);
+    janet_def_addflags(def);
     defindex = janetc_addfuncdef(c, def);
 
     /* Ensure enough slots for vararg function. */

src/core/state.h

@@ -34,6 +34,9 @@
 
 typedef struct JanetScratch JanetScratch;
 
+/* Top level dynamic bindings */
+extern JANET_THREAD_LOCAL JanetTable *janet_vm_top_dyns;
+
 /* Cache the core environment */
 extern JANET_THREAD_LOCAL JanetTable *janet_vm_core_env;
 
@@ -43,6 +46,7 @@ extern JANET_THREAD_LOCAL int janet_vm_stackn;
 /* The current running fiber on the current thread.
  * Set and unset by janet_run. */
 extern JANET_THREAD_LOCAL JanetFiber *janet_vm_fiber;
+extern JANET_THREAD_LOCAL JanetFiber *janet_vm_root_fiber;
 
 /* The current pointer to the inner most jmp_buf. The current
  * return point for panics. */
@@ -67,6 +71,7 @@ extern JANET_THREAD_LOCAL uint32_t janet_vm_cache_deleted;
 extern JANET_THREAD_LOCAL void *janet_vm_blocks;
 extern JANET_THREAD_LOCAL size_t janet_vm_gc_interval;
 extern JANET_THREAD_LOCAL size_t janet_vm_next_collection;
+extern JANET_THREAD_LOCAL size_t janet_vm_block_count;
 extern JANET_THREAD_LOCAL int janet_vm_gc_suspend;
 
 /* GC roots */
@@ -79,6 +84,17 @@ extern JANET_THREAD_LOCAL JanetScratch **janet_scratch_mem;
 extern JANET_THREAD_LOCAL size_t janet_scratch_cap;
 extern JANET_THREAD_LOCAL size_t janet_scratch_len;
 
+/* Recursionless traversal of data structures */
+typedef struct {
+    JanetGCObject *self;
+    JanetGCObject *other;
+    int32_t index;
+    int32_t index2;
+} JanetTraversalNode;
+extern JANET_THREAD_LOCAL JanetTraversalNode *janet_vm_traversal;
+extern JANET_THREAD_LOCAL JanetTraversalNode *janet_vm_traversal_top;
+extern JANET_THREAD_LOCAL JanetTraversalNode *janet_vm_traversal_base;
+
 /* Setup / teardown */
 #ifdef JANET_THREADS
 void janet_threads_init(void);

src/core/string.c

@@ -62,7 +62,7 @@ int janet_string_compare(const uint8_t *lhs, const uint8_t *rhs) {
     int32_t ylen = janet_string_length(rhs);
     int32_t len = xlen > ylen ? ylen : xlen;
     int res = memcmp(lhs, rhs, len);
-    if (res) return res;
+    if (res) return res > 0 ? 1 : -1;
     if (xlen == ylen) return 0;
     return xlen < ylen ? -1 : 1;
 }
@@ -176,6 +176,18 @@ static Janet cfun_string_slice(int32_t argc, Janet *argv) {
     return janet_stringv(view.bytes + range.start, range.end - range.start);
 }
 
+static Janet cfun_symbol_slice(int32_t argc, Janet *argv) {
+    JanetByteView view = janet_getbytes(argv, 0);
+    JanetRange range = janet_getslice(argc, argv);
+    return janet_symbolv(view.bytes + range.start, range.end - range.start);
+}
+
+static Janet cfun_keyword_slice(int32_t argc, Janet *argv) {
+    JanetByteView view = janet_getbytes(argv, 0);
+    JanetRange range = janet_getslice(argc, argv);
+    return janet_keywordv(view.bytes + range.start, range.end - range.start);
+}
+
 static Janet cfun_string_repeat(int32_t argc, Janet *argv) {
     janet_fixarity(argc, 2);
     JanetByteView view = janet_getbytes(argv, 0);
@@ -529,6 +541,16 @@ static const JanetReg string_cfuns[] = {
              "from the end of the string. Note that index -1 is synonymous with "
              "index (length bytes) to allow a full negative slice range. ")
     },
+    {
+        "keyword/slice", cfun_keyword_slice,
+        JDOC("(keyword/slice bytes &opt start end)\n\n"
+             "Same a string/slice, but returns a keyword.")
+    },
+    {
+        "symbol/slice", cfun_symbol_slice,
+        JDOC("(symbol/slice bytes &opt start end)\n\n"
+             "Same a string/slice, but returns a symbol.")
+    },
     {
         "string/repeat", cfun_string_repeat,
         JDOC("(string/repeat bytes n)\n\n"
@@ -542,7 +564,7 @@ static const JanetReg string_cfuns[] = {
     {
         "string/from-bytes", cfun_string_frombytes,
         JDOC("(string/from-bytes & byte-vals)\n\n"
-             "Creates a string from integer params with byte values. All integers "
+             "Creates a string from integer parameters with byte values. All integers "
              "will be coerced to the range of 1 byte 0-255.")
     },
     {
@@ -573,7 +595,7 @@ static const JanetReg string_cfuns[] = {
     },
     {
         "string/find-all", cfun_string_findall,
-        JDOC("(string/find patt str)\n\n"
+        JDOC("(string/find-all patt str)\n\n"
              "Searches for all instances of pattern patt in string "
              "str. Returns an array of all indices of found patterns. Overlapping "
              "instances of the pattern are not counted, meaning a byte in string "
@@ -627,7 +649,7 @@ static const JanetReg string_cfuns[] = {
     {
         "string/format", cfun_string_format,
         JDOC("(string/format format & values)\n\n"
-             "Similar to snprintf, but specialized for operating with janet. Returns "
+             "Similar to snprintf, but specialized for operating with Janet values. Returns "
              "a new string.")
     },
     {

src/core/strtod.c

@@ -208,9 +208,9 @@ static double convert(
 
     /* Approximate exponent in base 2 of mant and exponent. This should get us a good estimate of the final size of the
      * number, within * 2^32 or so. */
-    int32_t mant_exp2_approx = mant->n * 32 + 16;
-    int32_t exp_exp2_approx = (int32_t)(floor(log2(base) * exponent));
-    int32_t exp2_approx = mant_exp2_approx + exp_exp2_approx;
+    int64_t mant_exp2_approx = mant->n * 32 + 16;
+    int64_t exp_exp2_approx = (int64_t)(floor(log2(base) * exponent));
+    int64_t exp2_approx = mant_exp2_approx + exp_exp2_approx;
 
     /* Short circuit zero, huge, and small numbers. We use the exponent range of valid IEEE754 doubles (-1022, 1023)
      * with a healthy buffer to allow for inaccuracies in the approximation and denormailzed numbers. */

src/core/struct.c

@@ -123,7 +123,8 @@ void janet_struct_put(JanetKV *st, Janet key, Janet value) {
                 dist = otherdist;
                 hash = otherhash;
             } else if (status == 0) {
-                /* A key was added to the struct more than once */
+                /* A key was added to the struct more than once - replace old value */
+                kv->value = value;
                 return;
             }
         }
@@ -166,51 +167,3 @@ JanetTable *janet_struct_to_table(const JanetKV *st) {
     }
     return table;
 }
-
-/* Check if two structs are equal */
-int janet_struct_equal(const JanetKV *lhs, const JanetKV *rhs) {
-    int32_t index;
-    int32_t llen = janet_struct_capacity(lhs);
-    int32_t rlen = janet_struct_capacity(rhs);
-    int32_t lhash = janet_struct_hash(lhs);
-    int32_t rhash = janet_struct_hash(rhs);
-    if (llen != rlen)
-        return 0;
-    if (lhash != rhash)
-        return 0;
-    for (index = 0; index < llen; index++) {
-        const JanetKV *l = lhs + index;
-        const JanetKV *r = rhs + index;
-        if (!janet_equals(l->key, r->key))
-            return 0;
-        if (!janet_equals(l->value, r->value))
-            return 0;
-    }
-    return 1;
-}
-
-/* Compare structs */
-int janet_struct_compare(const JanetKV *lhs, const JanetKV *rhs) {
-    int32_t i;
-    int32_t lhash = janet_struct_hash(lhs);
-    int32_t rhash = janet_struct_hash(rhs);
-    int32_t llen = janet_struct_capacity(lhs);
-    int32_t rlen = janet_struct_capacity(rhs);
-    if (llen < rlen)
-        return -1;
-    if (llen > rlen)
-        return 1;
-    if (lhash < rhash)
-        return -1;
-    if (lhash > rhash)
-        return 1;
-    for (i = 0; i < llen; ++i) {
-        const JanetKV *l = lhs + i;
-        const JanetKV *r = rhs + i;
-        int comp = janet_compare(l->key, r->key);
-        if (comp != 0) return comp;
-        comp = janet_compare(l->value, r->value);
-        if (comp != 0) return comp;
-    }
-    return 0;
-}

src/core/table.c

@@ -173,7 +173,7 @@ Janet janet_table_rawget(JanetTable *t, Janet key) {
 Janet janet_table_remove(JanetTable *t, Janet key) {
     JanetKV *bucket = janet_table_find(t, key);
     if (NULL != bucket && !janet_checktype(bucket->key, JANET_NIL)) {
-        Janet ret = bucket->key;
+        Janet ret = bucket->value;
         t->count--;
         t->deleted++;
         bucket->key = janet_wrap_nil();

src/core/thread.c

@@ -66,9 +66,15 @@ struct JanetMailbox {
     JanetBuffer messages[];
 };
 
+#define JANET_THREAD_HEAVYWEIGHT 0x1
+#define JANET_THREAD_ABSTRACTS 0x2
+#define JANET_THREAD_CFUNCTIONS 0x4
+static const char janet_thread_flags[] = "hac";
+
 typedef struct {
     JanetMailbox *original;
     JanetMailbox *newbox;
+    uint64_t flags;
 } JanetMailboxPair;
 
 static JANET_THREAD_LOCAL JanetMailbox *janet_vm_mailbox = NULL;
@@ -175,7 +181,7 @@ static int thread_mark(void *p, size_t size) {
     return 0;
 }
 
-static JanetMailboxPair *make_mailbox_pair(JanetMailbox *original) {
+static JanetMailboxPair *make_mailbox_pair(JanetMailbox *original, uint64_t flags) {
     JanetMailboxPair *pair = malloc(sizeof(JanetMailboxPair));
     if (NULL == pair) {
         JANET_OUT_OF_MEMORY;
@@ -183,6 +189,7 @@ static JanetMailboxPair *make_mailbox_pair(JanetMailbox *original) {
     pair->original = original;
     janet_mailbox_ref(original, 1);
     pair->newbox = janet_mailbox_create(1, 16);
+    pair->flags = flags;
     return pair;
 }
 
@@ -227,7 +234,7 @@ static void janet_waiter_init(JanetWaiter *waiter, double sec) {
     if (waiter->timedwait) {
         /* N seconds -> timespec of (now + sec) */
         struct timespec now;
-        clock_gettime(CLOCK_REALTIME, &now);
+        janet_gettime(&now);
         time_t tvsec = (time_t) floor(sec);
         long tvnsec = (long) floor(1000000000.0 * (sec - ((double) tvsec)));
         tvsec += now.tv_sec;
@@ -329,7 +336,7 @@ int janet_thread_send(JanetThread *thread, Janet msg, double timeout) {
         msgbuf->count = 0;
 
         /* Start panic zone */
-        janet_marshal(msgbuf, msg, thread->encode, 0);
+        janet_marshal(msgbuf, msg, thread->encode, JANET_MARSHAL_UNSAFE);
         /* End panic zone */
 
         mailbox->messageNext = (mailbox->messageNext + 1) % mailbox->messageCapacity;
@@ -368,8 +375,12 @@ int janet_thread_receive(Janet *msg_out, double timeout) {
 
             /* Handle errors */
             if (setjmp(buf)) {
-                /* Cleanup jmp_buf, keep lock */
+                /* Cleanup jmp_buf, return error.
+                 * Do not ignore bad messages as before. */
                 janet_vm_jmp_buf = old_buf;
+                *msg_out = *janet_vm_return_reg;
+                janet_mailbox_unlock(mailbox);
+                return 2;
             } else {
                 JanetBuffer *msgbuf = mailbox->messages + mailbox->messageFirst;
                 mailbox->messageCount--;
@@ -379,7 +390,7 @@ int janet_thread_receive(Janet *msg_out, double timeout) {
                 const uint8_t *nextItem = NULL;
                 Janet item = janet_unmarshal(
                                  msgbuf->data, msgbuf->count,
-                                 0, janet_thread_get_decode(), &nextItem);
+                                 JANET_MARSHAL_UNSAFE, janet_thread_get_decode(), &nextItem);
                 *msg_out = item;
 
                 /* Cleanup */
@@ -404,7 +415,6 @@ int janet_thread_receive(Janet *msg_out, double timeout) {
             return 1;
         }
     }
-
 }
 
 static int janet_thread_getter(void *p, Janet key, Janet *out);
@@ -442,16 +452,44 @@ static int thread_worker(JanetMailboxPair *pair) {
     janet_init();
 
     /* Get dictionaries for default encode/decode */
-    JanetTable *encode = janet_get_core_table("make-image-dict");
+    JanetTable *encode;
+    if (pair->flags & JANET_THREAD_HEAVYWEIGHT) {
+        encode = janet_get_core_table("make-image-dict");
+    } else {
+        encode = NULL;
+        janet_vm_thread_decode = janet_table(0);
+        janet_gcroot(janet_wrap_table(janet_vm_thread_decode));
+    }
 
     /* Create parent thread */
     JanetThread *parent = janet_make_thread(pair->original, encode);
     Janet parentv = janet_wrap_abstract(parent);
 
+    /* Unmarshal the abstract registry */
+    if (pair->flags & JANET_THREAD_ABSTRACTS) {
+        Janet reg;
+        int status = janet_thread_receive(&reg, INFINITY);
+        if (status) goto error;
+        if (!janet_checktype(reg, JANET_TABLE)) goto error;
+        janet_gcunroot(janet_wrap_table(janet_vm_abstract_registry));
+        janet_vm_abstract_registry = janet_unwrap_table(reg);
+        janet_gcroot(janet_wrap_table(janet_vm_abstract_registry));
+    }
+
+    /* Unmarshal the normal registry */
+    if (pair->flags & JANET_THREAD_CFUNCTIONS) {
+        Janet reg;
+        int status = janet_thread_receive(&reg, INFINITY);
+        if (status) goto error;
+        if (!janet_checktype(reg, JANET_TABLE)) goto error;
+        janet_gcunroot(janet_wrap_table(janet_vm_registry));
+        janet_vm_registry = janet_unwrap_table(reg);
+        janet_gcroot(janet_wrap_table(janet_vm_registry));
+    }
+
     /* Unmarshal the function */
     Janet funcv;
     int status = janet_thread_receive(&funcv, INFINITY);
-
     if (status) goto error;
     if (!janet_checktype(funcv, JANET_FUNCTION)) goto error;
     JanetFunction *func = janet_unwrap_function(funcv);
@@ -464,12 +502,20 @@ static int thread_worker(JanetMailboxPair *pair) {
     /* Call function */
     Janet argv[1] = { parentv };
     fiber = janet_fiber(func, 64, 1, argv);
+    if (pair->flags & JANET_THREAD_HEAVYWEIGHT) {
+        fiber->env = janet_table(0);
+        fiber->env->proto = janet_core_env(NULL);
+    }
     JanetSignal sig = janet_continue(fiber, janet_wrap_nil(), &out);
-    if (sig != JANET_SIGNAL_OK) {
+    if (sig != JANET_SIGNAL_OK && sig < JANET_SIGNAL_USER0) {
         janet_eprintf("in thread %v: ", janet_wrap_abstract(janet_make_thread(pair->newbox, encode)));
         janet_stacktrace(fiber, out);
     }
 
+#ifdef JANET_NET
+    janet_loop();
+#endif
+
     /* Normal exit */
     destroy_mailbox_pair(pair);
     janet_deinit();
@@ -554,22 +600,40 @@ static Janet cfun_thread_current(int32_t argc, Janet *argv) {
 }
 
 static Janet cfun_thread_new(int32_t argc, Janet *argv) {
-    janet_arity(argc, 1, 2);
+    janet_arity(argc, 1, 3);
     /* Just type checking */
     janet_getfunction(argv, 0);
     int32_t cap = janet_optinteger(argv, argc, 1, 10);
     if (cap < 1 || cap > UINT16_MAX) {
         janet_panicf("bad slot #1, expected integer in range [1, 65535], got %d", cap);
     }
-    JanetTable *encode = janet_get_core_table("make-image-dict");
+    uint64_t flags = argc >= 3 ? janet_getflags(argv, 2, janet_thread_flags) : JANET_THREAD_ABSTRACTS;
+    JanetTable *encode;
+    if (flags & JANET_THREAD_HEAVYWEIGHT) {
+        encode = janet_get_core_table("make-image-dict");
+    } else {
+        encode = NULL;
+    }
 
-    JanetMailboxPair *pair = make_mailbox_pair(janet_vm_mailbox);
+    JanetMailboxPair *pair = make_mailbox_pair(janet_vm_mailbox, flags);
     JanetThread *thread = janet_make_thread(pair->newbox, encode);
     if (janet_thread_start_child(pair)) {
         destroy_mailbox_pair(pair);
         janet_panic("could not start thread");
     }
 
+    if (flags & JANET_THREAD_ABSTRACTS) {
+        if (janet_thread_send(thread, janet_wrap_table(janet_vm_abstract_registry), INFINITY)) {
+            janet_panic("could not send abstract registry to thread");
+        }
+    }
+
+    if (flags & JANET_THREAD_CFUNCTIONS) {
+        if (janet_thread_send(thread, janet_wrap_table(janet_vm_registry), INFINITY)) {
+            janet_panic("could not send registry to thread");
+        }
+    }
+
     /* If thread started, send the worker function. */
     if (janet_thread_send(thread, argv[0], INFINITY)) {
         janet_panicf("could not send worker function %v to thread", argv[0]);
@@ -603,6 +667,8 @@ static Janet cfun_thread_receive(int32_t argc, Janet *argv) {
             break;
         case 1:
             janet_panicf("timeout after %f seconds", wait);
+        case 2:
+            janet_panicf("failed to receive message: %v", out);
     }
     return out;
 }
@@ -614,6 +680,18 @@ static Janet cfun_thread_close(int32_t argc, Janet *argv) {
     return janet_wrap_nil();
 }
 
+static Janet cfun_thread_exit(int32_t argc, Janet *argv) {
+    (void) argv;
+    janet_arity(argc, 0, 1);
+#if defined(JANET_WINDOWS)
+    int32_t flag = janet_optinteger(argv, argc, 0, 0);
+    ExitThread(flag);
+#else
+    pthread_exit(NULL);
+#endif
+    return janet_wrap_nil();
+}
+
 static const JanetMethod janet_thread_methods[] = {
     {"send", cfun_thread_send},
     {"close", cfun_thread_close},
@@ -634,10 +712,14 @@ static const JanetReg threadlib_cfuns[] = {
     },
     {
         "thread/new", cfun_thread_new,
-        JDOC("(thread/new func &opt capacity)\n\n"
+        JDOC("(thread/new func &opt capacity flags)\n\n"
              "Start a new thread that will start immediately. "
              "If capacity is provided, that is how many messages can be stored in the thread's mailbox before blocking senders. "
              "The capacity must be between 1 and 65535 inclusive, and defaults to 10. "
+             "Can optionally provide flags to the new thread - supported flags are:\n"
+             "\t:h - Start a heavyweight thread. This loads the core environment by default, so may use more memory initially. Messages may compress better, though.\n"
+             "\t:a - Allow sending over registered abstract types to the new thread\n"
+             "\t:c - Send over cfunction information to the new thread.\n"
              "Returns a handle to the new thread.")
     },
     {
@@ -658,6 +740,12 @@ static const JanetReg threadlib_cfuns[] = {
              "Close a thread, unblocking it and ending communication with it. Note that closing "
              "a thread is idempotent and does not cancel the thread's operation. Returns nil.")
     },
+    {
+        "thread/exit", cfun_thread_exit,
+        JDOC("(thread/exit &opt code)\n\n"
+             "Exit from the current thread. If no more threads are running, ends the process, but otherwise does "
+             "not end the current process.")
+    },
     {NULL, NULL, NULL}
 };
 

src/core/tuple.c

@@ -53,45 +53,6 @@ const Janet *janet_tuple_n(const Janet *values, int32_t n) {
     return janet_tuple_end(t);
 }
 
-/* Check if two tuples are equal */
-int janet_tuple_equal(const Janet *lhs, const Janet *rhs) {
-    int32_t index;
-    int32_t llen = janet_tuple_length(lhs);
-    int32_t rlen = janet_tuple_length(rhs);
-    int32_t lhash = janet_tuple_hash(lhs);
-    int32_t rhash = janet_tuple_hash(rhs);
-    if (lhash == 0)
-        lhash = janet_tuple_hash(lhs) = janet_array_calchash(lhs, llen);
-    if (rhash == 0)
-        rhash = janet_tuple_hash(rhs) = janet_array_calchash(rhs, rlen);
-    if (lhash != rhash)
-        return 0;
-    if (llen != rlen)
-        return 0;
-    for (index = 0; index < llen; index++) {
-        if (!janet_equals(lhs[index], rhs[index]))
-            return 0;
-    }
-    return 1;
-}
-
-/* Compare tuples */
-int janet_tuple_compare(const Janet *lhs, const Janet *rhs) {
-    int32_t i;
-    int32_t llen = janet_tuple_length(lhs);
-    int32_t rlen = janet_tuple_length(rhs);
-    int32_t count = llen < rlen ? llen : rlen;
-    for (i = 0; i < count; ++i) {
-        int comp = janet_compare(lhs[i], rhs[i]);
-        if (comp != 0) return comp;
-    }
-    if (llen < rlen)
-        return -1;
-    else if (llen > rlen)
-        return 1;
-    return 0;
-}
-
 /* C Functions */
 
 static Janet cfun_tuple_brackets(int32_t argc, Janet *argv) {

src/core/typedarray.c

@@ -275,6 +275,24 @@ static void ta_setter(void *p, Janet key, Janet value) {
     }
 }
 
+static Janet ta_view_next(void *p, Janet key) {
+    JanetTArrayView *view = p;
+    if (janet_checktype(key, JANET_NIL)) {
+        if (view->size > 0) {
+            return janet_wrap_number(0);
+        } else {
+            return janet_wrap_nil();
+        }
+    }
+    if (!janet_checksize(key)) janet_panic("expected size as key");
+    size_t index = (size_t) janet_unwrap_number(key);
+    index++;
+    if (index < view->size) {
+        return janet_wrap_number((double) index);
+    }
+    return janet_wrap_nil();
+}
+
 const JanetAbstractType janet_ta_view_type = {
     "ta/view",
     NULL,
@@ -283,7 +301,11 @@ const JanetAbstractType janet_ta_view_type = {
     ta_setter,
     ta_view_marshal,
     ta_view_unmarshal,
-    JANET_ATEND_UNMARSHAL
+    NULL,
+    NULL,
+    NULL,
+    ta_view_next,
+    JANET_ATEND_NEXT
 };
 
 JanetTArrayBuffer *janet_tarray_buffer(size_t size) {
@@ -354,18 +376,29 @@ static Janet cfun_typed_array_new(int32_t argc, Janet *argv) {
     if (argc > 3)
         offset = janet_getsize(argv, 3);
     if (argc > 4) {
-        if (!janet_checktype(argv[4], JANET_ABSTRACT)) {
-            janet_panicf("bad slot #%d, expected ta/view|ta/buffer, got %v",
-                         4, argv[4]);
-        }
-        void *p = janet_unwrap_abstract(argv[4]);
-        if (janet_abstract_type(p) == &janet_ta_view_type) {
-            JanetTArrayView *view = (JanetTArrayView *)p;
-            offset = (view->buffer->data - view->as.u8) + offset * ta_type_sizes[view->type];
-            stride *= view->stride;
-            buffer = view->buffer;
+        int32_t blen;
+        const uint8_t *bytes;
+        if (janet_bytes_view(argv[4], &bytes, &blen)) {
+            buffer = janet_abstract(&janet_ta_buffer_type, sizeof(JanetTArrayBuffer));
+            ta_buffer_init(buffer, (size_t) blen);
+            memcpy(buffer->data, bytes, blen);
         } else {
-            buffer = p;
+            if (!janet_checktype(argv[4], JANET_ABSTRACT)) {
+                janet_panicf("bad slot #%d, expected ta/view|ta/buffer, got %v",
+                             4, argv[4]);
+            }
+            void *p = janet_unwrap_abstract(argv[4]);
+            if (janet_abstract_type(p) == &janet_ta_view_type) {
+                JanetTArrayView *view = (JanetTArrayView *)p;
+                offset = (view->buffer->data - view->as.u8) + offset * ta_type_sizes[view->type];
+                stride *= view->stride;
+                buffer = view->buffer;
+            } else if (janet_abstract_type(p) == &janet_ta_buffer_type) {
+                buffer = p;
+            } else {
+                janet_panicf("bad slot #%d, expected ta/view|ta/buffer, got %v",
+                             4, argv[4]);
+            }
         }
     }
     JanetTArrayView *view = janet_tarray_view(type, size, stride, offset, buffer);

src/core/util.c

@@ -26,6 +26,9 @@
 #include "util.h"
 #include "state.h"
 #include "gc.h"
+#ifdef JANET_WINDOWS
+#include <windows.h>
+#endif
 #endif
 
 #include <inttypes.h>
@@ -94,7 +97,7 @@ const char *const janet_status_names[16] = {
     "alive"
 };
 
-#ifdef JANET_NO_PRF
+#ifndef JANET_PRF
 
 int32_t janet_string_calchash(const uint8_t *str, int32_t len) {
     const uint8_t *end = str + len;
@@ -380,7 +383,7 @@ void janet_var(JanetTable *env, const char *name, Janet val, const char *doc) {
 }
 
 /* Load many cfunctions at once */
-void janet_cfuns(JanetTable *env, const char *regprefix, const JanetReg *cfuns) {
+static void _janet_cfuns_prefix(JanetTable *env, const char *regprefix, const JanetReg *cfuns, int defprefix) {
     uint8_t *longname_buffer = NULL;
     size_t prefixlen = 0;
     size_t bufsize = 0;
@@ -414,13 +417,29 @@ void janet_cfuns(JanetTable *env, const char *regprefix, const JanetReg *cfuns)
             name = janet_csymbolv(cfuns->name);
         }
         Janet fun = janet_wrap_cfunction(cfuns->cfun);
-        janet_def(env, cfuns->name, fun, cfuns->documentation);
+        if (defprefix) {
+            JanetTable *subt = janet_table(2);
+            janet_table_put(subt, janet_ckeywordv("value"), fun);
+            if (cfuns->documentation)
+                janet_table_put(subt, janet_ckeywordv("doc"), janet_cstringv(cfuns->documentation));
+            janet_table_put(env, name, janet_wrap_table(subt));
+        } else {
+            janet_def(env, cfuns->name, fun, cfuns->documentation);
+        }
         janet_table_put(janet_vm_registry, fun, name);
         cfuns++;
     }
     free(longname_buffer);
 }
 
+void janet_cfuns_prefix(JanetTable *env, const char *regprefix, const JanetReg *cfuns) {
+    _janet_cfuns_prefix(env, regprefix, cfuns, 1);
+}
+
+void janet_cfuns(JanetTable *env, const char *regprefix, const JanetReg *cfuns) {
+    _janet_cfuns_prefix(env, regprefix, cfuns, 0);
+}
+
 /* Abstract type introspection */
 
 void janet_register_abstract_type(const JanetAbstractType *at) {
@@ -558,8 +577,12 @@ int janet_checksize(Janet x) {
     if (!janet_checktype(x, JANET_NUMBER))
         return 0;
     double dval = janet_unwrap_number(x);
-    return dval == (double)((size_t) dval) &&
-           dval <= SIZE_MAX;
+    if (dval != (double)((size_t) dval)) return 0;
+    if (SIZE_MAX > JANET_INTMAX_INT64) {
+        return dval <= JANET_INTMAX_INT64;
+    } else {
+        return dval <= SIZE_MAX;
+    }
 }
 
 JanetTable *janet_get_core_table(const char *name) {
@@ -570,3 +593,40 @@ JanetTable *janet_get_core_table(const char *name) {
     if (!janet_checktype(out, JANET_TABLE)) return NULL;
     return janet_unwrap_table(out);
 }
+
+/* Clock shims for various platforms */
+#ifdef JANET_GETTIME
+/* For macos */
+#ifdef __MACH__
+#include <mach/clock.h>
+#include <mach/mach.h>
+#endif
+#ifdef JANET_WINDOWS
+int janet_gettime(struct timespec *spec) {
+    FILETIME ftime;
+    GetSystemTimeAsFileTime(&ftime);
+    int64_t wintime = (int64_t)(ftime.dwLowDateTime) | ((int64_t)(ftime.dwHighDateTime) << 32);
+    /* Windows epoch is January 1, 1601 apparently */
+    wintime -= 116444736000000000LL;
+    spec->tv_sec  = wintime / 10000000LL;
+    /* Resolution is 100 nanoseconds. */
+    spec->tv_nsec = wintime % 10000000LL * 100;
+    return 0;
+}
+#elif defined(__MACH__)
+int janet_gettime(struct timespec *spec) {
+    clock_serv_t cclock;
+    mach_timespec_t mts;
+    host_get_clock_service(mach_host_self(), CALENDAR_CLOCK, &cclock);
+    clock_get_time(cclock, &mts);
+    mach_port_deallocate(mach_task_self(), cclock);
+    spec->tv_sec = mts.tv_sec;
+    spec->tv_nsec = mts.tv_nsec;
+    return 0;
+}
+#else
+int janet_gettime(struct timespec *spec) {
+    return clock_gettime(CLOCK_REALTIME, spec);
+}
+#endif
+#endif

src/core/util.h

@@ -31,11 +31,16 @@
 #include <stdio.h>
 #include <errno.h>
 
+#if !defined(JANET_REDUCED_OS) || !defined(JANET_SINGLE_THREADED)
+#include <time.h>
+#define JANET_GETTIME
+#endif
+
 /* Handle runtime errors */
-#ifndef janet_exit
+#ifndef JANET_EXIT
 #include <stdio.h>
-#define janet_exit(m) do { \
-    printf("C runtime error at line %d in file %s: %s\n",\
+#define JANET_EXIT(m) do { \
+    fprintf(stderr, "C runtime error at line %d in file %s: %s\n",\
         __LINE__,\
         __FILE__,\
         (m));\
@@ -44,13 +49,13 @@
 #endif
 
 #define janet_assert(c, m) do { \
-    if (!(c)) janet_exit((m)); \
+    if (!(c)) JANET_EXIT((m)); \
 } while (0)
 
 /* What to do when out of memory */
 #ifndef JANET_OUT_OF_MEMORY
 #include <stdio.h>
-#define JANET_OUT_OF_MEMORY do { printf("janet out of memory\n"); exit(1); } while (0)
+#define JANET_OUT_OF_MEMORY do { fprintf(stderr, "janet out of memory\n"); exit(1); } while (0)
 #endif
 
 /* Omit docstrings in some builds */
@@ -75,6 +80,7 @@ Janet janet_dict_get(const JanetKV *buckets, int32_t cap, Janet key);
 void janet_memempty(JanetKV *mem, int32_t count);
 void *janet_memalloc_empty(int32_t count);
 JanetTable *janet_get_core_table(const char *name);
+void janet_def_addflags(JanetFuncDef *def);
 const void *janet_strbinsearch(
     const void *tab,
     size_t tabcount,
@@ -97,6 +103,11 @@ void janet_core_def(JanetTable *env, const char *name, Janet x, const void *p);
 void janet_core_cfuns(JanetTable *env, const char *regprefix, const JanetReg *cfuns);
 #endif
 
+/* Clock gettime */
+#ifdef JANET_GETTIME
+int janet_gettime(struct timespec *spec);
+#endif
+
 /* Initialize builtin libraries */
 void janet_lib_io(JanetTable *env);
 void janet_lib_math(JanetTable *env);
@@ -126,5 +137,10 @@ void janet_lib_inttypes(JanetTable *env);
 #ifdef JANET_THREADS
 void janet_lib_thread(JanetTable *env);
 #endif
+#ifdef JANET_NET
+void janet_lib_net(JanetTable *env);
+void janet_net_deinit(void);
+void janet_net_markloop(void);
+#endif
 
 #endif

src/core/value.c

@@ -23,9 +23,91 @@
 #ifndef JANET_AMALG
 #include "features.h"
 #include "util.h"
+#include "state.h"
+#include "gc.h"
 #include <janet.h>
 #endif
 
+JANET_THREAD_LOCAL JanetTraversalNode *janet_vm_traversal = NULL;
+JANET_THREAD_LOCAL JanetTraversalNode *janet_vm_traversal_top = NULL;
+JANET_THREAD_LOCAL JanetTraversalNode *janet_vm_traversal_base = NULL;
+
+static void push_traversal_node(void *lhs, void *rhs, int32_t index2) {
+    JanetTraversalNode node;
+    node.self = (JanetGCObject *) lhs;
+    node.other = (JanetGCObject *) rhs;
+    node.index = 0;
+    node.index2 = index2;
+    if (janet_vm_traversal + 1 >= janet_vm_traversal_top) {
+        size_t oldsize = janet_vm_traversal - janet_vm_traversal_base;
+        size_t newsize = 2 * oldsize + 1;
+        if (newsize < 128) {
+            newsize = 128;
+        }
+        JanetTraversalNode *tn = realloc(janet_vm_traversal_base, newsize * sizeof(JanetTraversalNode));
+        if (tn == NULL) {
+            JANET_OUT_OF_MEMORY;
+        }
+        janet_vm_traversal_base = tn;
+        janet_vm_traversal_top = janet_vm_traversal_base + newsize;
+        janet_vm_traversal = janet_vm_traversal_base + oldsize;
+    }
+    *(++janet_vm_traversal) = node;
+}
+
+/*
+ * Used for travsersing structs and tuples without recursion
+ * Returns:
+ * 0 - next node found
+ * 1 - early stop - lhs < rhs
+ * 2 - no next node found
+ * 3 - early stop - lhs > rhs
+ */
+static int traversal_next(Janet *x, Janet *y) {
+    JanetTraversalNode *t = janet_vm_traversal;
+    while (t && t > janet_vm_traversal_base) {
+        JanetGCObject *self = t->self;
+        JanetTupleHead *tself = (JanetTupleHead *)self;
+        JanetStructHead *sself = (JanetStructHead *)self;
+        JanetGCObject *other = t->other;
+        JanetTupleHead *tother = (JanetTupleHead *)other;
+        JanetStructHead *sother = (JanetStructHead *)other;
+        if ((self->flags & JANET_MEM_TYPEBITS) == JANET_MEMORY_TUPLE) {
+            /* Node is a tuple at index t->index */
+            if (t->index < tself->length && t->index < tother->length) {
+                int32_t index = t->index++;
+                *x = tself->data[index];
+                *y = tother->data[index];
+                janet_vm_traversal = t;
+                return 0;
+            }
+            if (t->index2 && tself->length != tother->length) {
+                return tself->length > tother->length ? 3 : 1;
+            }
+        } else {
+            /* Node is a struct at index t->index: if t->index2 is true, we should return the values. */
+            if (t->index2) {
+                t->index2 = 0;
+                int32_t index = t->index++;
+                *x = sself->data[index].value;
+                *y = sother->data[index].value;
+                janet_vm_traversal = t;
+                return 0;
+            }
+            for (int32_t i = t->index; i < sself->capacity; i++) {
+                t->index2 = 1;
+                *x = sself->data[t->index].key;
+                *y = sother->data[t->index].key;
+                janet_vm_traversal = t;
+                return 0;
+            }
+        }
+        t--;
+    }
+    janet_vm_traversal = t;
+    return 2;
+}
+
 /*
  * Define a number of functions that can be used internally on ANY Janet.
  */
@@ -111,41 +193,51 @@ static int janet_compare_abstract(JanetAbstract xx, JanetAbstract yy) {
     return xt->compare(xx, yy);
 }
 
-/* Check if two values are equal. This is strict equality with no conversion. */
 int janet_equals(Janet x, Janet y) {
-    int result = 0;
-    if (janet_type(x) != janet_type(y)) {
-        result = 0;
-    } else {
+    janet_vm_traversal = janet_vm_traversal_base;
+    do {
+        if (janet_type(x) != janet_type(y)) return 0;
         switch (janet_type(x)) {
             case JANET_NIL:
-                result = 1;
                 break;
             case JANET_BOOLEAN:
-                result = (janet_unwrap_boolean(x) == janet_unwrap_boolean(y));
+                if (janet_unwrap_boolean(x) != janet_unwrap_boolean(y)) return 0;
                 break;
             case JANET_NUMBER:
-                result = (janet_unwrap_number(x) == janet_unwrap_number(y));
+                if (janet_unwrap_number(x) != janet_unwrap_number(y)) return 0;
                 break;
             case JANET_STRING:
-                result = janet_string_equal(janet_unwrap_string(x), janet_unwrap_string(y));
-                break;
-            case JANET_TUPLE:
-                result = janet_tuple_equal(janet_unwrap_tuple(x), janet_unwrap_tuple(y));
-                break;
-            case JANET_STRUCT:
-                result = janet_struct_equal(janet_unwrap_struct(x), janet_unwrap_struct(y));
+                if (!janet_string_equal(janet_unwrap_string(x), janet_unwrap_string(y))) return 0;
                 break;
             case JANET_ABSTRACT:
-                result = !janet_compare_abstract(janet_unwrap_abstract(x), janet_unwrap_abstract(y));
+                if (janet_compare_abstract(janet_unwrap_abstract(x), janet_unwrap_abstract(y))) return 0;
                 break;
             default:
-                /* compare pointers */
-                result = (janet_unwrap_pointer(x) == janet_unwrap_pointer(y));
+                if (janet_unwrap_pointer(x) != janet_unwrap_pointer(y)) return 0;
                 break;
+            case JANET_TUPLE: {
+                const Janet *t1 = janet_unwrap_tuple(x);
+                const Janet *t2 = janet_unwrap_tuple(y);
+                if (t1 == t2) break;
+                if (janet_tuple_hash(t1) != janet_tuple_hash(t2)) return 0;
+                if (janet_tuple_length(t1) != janet_tuple_length(t2)) return 0;
+                push_traversal_node(janet_tuple_head(t1), janet_tuple_head(t2), 0);
+                break;
+            }
+            break;
+            case JANET_STRUCT: {
+                const JanetKV *s1 = janet_unwrap_struct(x);
+                const JanetKV *s2 = janet_unwrap_struct(y);
+                if (s1 == s2) break;
+                if (janet_struct_hash(s1) != janet_struct_hash(s2)) return 0;
+                if (janet_struct_length(s1) != janet_struct_length(s2)) return 0;
+                push_traversal_node(janet_struct_head(s1), janet_struct_head(s2), 0);
+                break;
+            }
+            break;
         }
-    }
-    return result;
+    } while (!traversal_next(&x, &y));
+    return 1;
 }
 
 /* Computes a hash value for a function */
@@ -201,38 +293,71 @@ int32_t janet_hash(Janet x) {
  * If y is less, returns 1. All types are comparable
  * and should have strict ordering, excepts NaNs. */
 int janet_compare(Janet x, Janet y) {
-    if (janet_type(x) == janet_type(y)) {
-        switch (janet_type(x)) {
+    janet_vm_traversal = janet_vm_traversal_base;
+    int status;
+    do {
+        JanetType tx = janet_type(x);
+        JanetType ty = janet_type(y);
+        if (tx != ty) return tx < ty ? -1 : 1;
+        switch (tx) {
             case JANET_NIL:
-                return 0;
-            case JANET_BOOLEAN:
-                return janet_unwrap_boolean(x) - janet_unwrap_boolean(y);
+                break;
+            case JANET_BOOLEAN: {
+                int diff = janet_unwrap_boolean(x) - janet_unwrap_boolean(y);
+                if (diff) return diff;
+                break;
+            }
             case JANET_NUMBER: {
                 double xx = janet_unwrap_number(x);
                 double yy = janet_unwrap_number(y);
-                return xx == yy
-                       ? 0
-                       : (xx < yy) ? -1 : 1;
+                if (xx == yy) {
+                    break;
+                } else {
+                    return (xx < yy) ? -1 : 1;
+                }
             }
             case JANET_STRING:
             case JANET_SYMBOL:
-            case JANET_KEYWORD:
-                return janet_string_compare(janet_unwrap_string(x), janet_unwrap_string(y));
-            case JANET_TUPLE:
-                return janet_tuple_compare(janet_unwrap_tuple(x), janet_unwrap_tuple(y));
-            case JANET_STRUCT:
-                return janet_struct_compare(janet_unwrap_struct(x), janet_unwrap_struct(y));
-            case JANET_ABSTRACT:
-                return janet_compare_abstract(janet_unwrap_abstract(x), janet_unwrap_abstract(y));
-            default:
-                if (janet_unwrap_string(x) == janet_unwrap_string(y)) {
-                    return 0;
+            case JANET_KEYWORD: {
+                int diff = janet_string_compare(janet_unwrap_string(x), janet_unwrap_string(y));
+                if (diff) return diff;
+                break;
+            }
+            case JANET_ABSTRACT: {
+                int diff = janet_compare_abstract(janet_unwrap_abstract(x), janet_unwrap_abstract(y));
+                if (diff) return diff;
+                break;
+            }
+            default: {
+                if (janet_unwrap_pointer(x) == janet_unwrap_pointer(y)) {
+                    break;
                 } else {
-                    return janet_unwrap_string(x) > janet_unwrap_string(y) ? 1 : -1;
+                    return janet_unwrap_pointer(x) > janet_unwrap_pointer(y) ? 1 : -1;
                 }
+            }
+            case JANET_TUPLE: {
+                const Janet *lhs = janet_unwrap_tuple(x);
+                const Janet *rhs = janet_unwrap_tuple(y);
+                push_traversal_node(janet_tuple_head(lhs), janet_tuple_head(rhs), 1);
+                break;
+            }
+            case JANET_STRUCT: {
+                const JanetKV *lhs = janet_unwrap_struct(x);
+                const JanetKV *rhs = janet_unwrap_struct(y);
+                int32_t llen = janet_struct_capacity(lhs);
+                int32_t rlen = janet_struct_capacity(rhs);
+                int32_t lhash = janet_struct_hash(lhs);
+                int32_t rhash = janet_struct_hash(rhs);
+                if (llen < rlen) return -1;
+                if (llen > rlen) return 1;
+                if (lhash < rhash) return -1;
+                if (lhash > rhash) return 1;
+                push_traversal_node(janet_struct_head(lhs), janet_struct_head(rhs), 0);
+                break;
+            }
         }
-    }
-    return (janet_type(x) < janet_type(y)) ? -1 : 1;
+    } while (!(status = traversal_next(&x, &y)));
+    return status - 2;
 }
 
 static int32_t getter_checkint(Janet key, int32_t max) {

src/core/vm.c

@@ -33,11 +33,13 @@
 #include <math.h>
 
 /* VM state */
+JANET_THREAD_LOCAL JanetTable *janet_vm_top_dyns;
 JANET_THREAD_LOCAL JanetTable *janet_vm_core_env;
 JANET_THREAD_LOCAL JanetTable *janet_vm_registry;
 JANET_THREAD_LOCAL JanetTable *janet_vm_abstract_registry;
 JANET_THREAD_LOCAL int janet_vm_stackn = 0;
 JANET_THREAD_LOCAL JanetFiber *janet_vm_fiber = NULL;
+JANET_THREAD_LOCAL JanetFiber *janet_vm_root_fiber = NULL;
 JANET_THREAD_LOCAL Janet *janet_vm_return_reg = NULL;
 JANET_THREAD_LOCAL jmp_buf *janet_vm_jmp_buf = NULL;
 
@@ -89,8 +91,8 @@ JANET_THREAD_LOCAL jmp_buf *janet_vm_jmp_buf = NULL;
     func = janet_stack_frame(stack)->func; \
 } while (0)
 #define vm_return(sig, val) do { \
-    vm_commit(); \
     janet_vm_return_reg[0] = (val); \
+    vm_commit(); \
     return (sig); \
 } while (0)
 
@@ -107,13 +109,13 @@ JANET_THREAD_LOCAL jmp_buf *janet_vm_jmp_buf = NULL;
 #define vm_assert_type(X, T) do { \
     if (!(janet_checktype((X), (T)))) { \
         vm_commit(); \
-        janet_panicf("expected %T, got %t", (1 << (T)), (X)); \
+        janet_panicf("expected %T, got %v", (1 << (T)), (X)); \
     } \
 } while (0)
 #define vm_assert_types(X, TS) do { \
     if (!(janet_checktypes((X), (TS)))) { \
         vm_commit(); \
-        janet_panicf("expected %T, got %t", (TS), (X)); \
+        janet_panicf("expected %T, got %v", (TS), (X)); \
     } \
 } while (0)
 
@@ -290,6 +292,10 @@ static Janet janet_binop_call(const char *lmethod, const char *rmethod, Janet lh
     }
 }
 
+/* Forward declaration */
+static JanetSignal janet_check_can_resume(JanetFiber *fiber, Janet *out);
+static JanetSignal janet_continue_no_check(JanetFiber *fiber, Janet in, Janet *out);
+
 /* Interpreter main loop */
 static JanetSignal run_vm(JanetFiber *fiber, Janet in) {
 
@@ -368,8 +374,8 @@ static JanetSignal run_vm(JanetFiber *fiber, Janet in) {
         &&label_JOP_GREATER_THAN_EQUAL,
         &&label_JOP_LESS_THAN_EQUAL,
         &&label_JOP_NEXT,
-        &&label_unknown_op,
-        &&label_unknown_op,
+        &&label_JOP_NOT_EQUALS,
+        &&label_JOP_NOT_EQUALS_IMMEDIATE,
         &&label_unknown_op,
         &&label_unknown_op,
         &&label_unknown_op,
@@ -782,6 +788,14 @@ static JanetSignal run_vm(JanetFiber *fiber, Janet in) {
     stack[A] = janet_wrap_boolean(janet_unwrap_integer(stack[B]) == CS);
     vm_pcnext();
 
+    VM_OP(JOP_NOT_EQUALS)
+    stack[A] = janet_wrap_boolean(!janet_equals(stack[B], stack[C]));
+    vm_pcnext();
+
+    VM_OP(JOP_NOT_EQUALS_IMMEDIATE)
+    stack[A] = janet_wrap_boolean(janet_unwrap_integer(stack[B]) != CS);
+    vm_pcnext();
+
     VM_OP(JOP_COMPARE)
     stack[A] = janet_wrap_integer(janet_compare(stack[B], stack[C]));
     vm_pcnext();
@@ -824,7 +838,8 @@ static JanetSignal run_vm(JanetFiber *fiber, Janet in) {
         vm_assert(func->def->environments_length > eindex, "invalid upvalue environment");
         env = func->envs[eindex];
         vm_assert(env->length > vindex, "invalid upvalue index");
-        if (env->offset) {
+        vm_assert(janet_env_valid(env), "invalid upvalue environment");
+        if (env->offset > 0) {
             /* On stack */
             stack[A] = env->as.fiber->data[env->offset + vindex];
         } else {
@@ -841,7 +856,8 @@ static JanetSignal run_vm(JanetFiber *fiber, Janet in) {
         vm_assert(func->def->environments_length > eindex, "invalid upvalue environment");
         env = func->envs[eindex];
         vm_assert(env->length > vindex, "invalid upvalue index");
-        if (env->offset) {
+        vm_assert(janet_env_valid(env), "invalid upvalue environment");
+        if (env->offset > 0) {
             env->as.fiber->data[env->offset + vindex] = stack[A];
         } else {
             env->as.values[vindex] = stack[A];
@@ -904,7 +920,7 @@ static JanetSignal run_vm(JanetFiber *fiber, Janet in) {
         if (janet_indexed_view(stack[D], &vals, &len)) {
             janet_fiber_pushn(fiber, vals, len);
         } else {
-            janet_panicf("expected %T, got %t", JANET_TFLAG_INDEXED, stack[D]);
+            janet_panicf("expected %T, got %v", JANET_TFLAG_INDEXED, stack[D]);
         }
     }
     stack = fiber->data + fiber->frame;
@@ -922,7 +938,7 @@ static JanetSignal run_vm(JanetFiber *fiber, Janet in) {
         if (janet_checktype(callee, JANET_FUNCTION)) {
             func = janet_unwrap_function(callee);
             if (func->gc.flags & JANET_FUNCFLAG_TRACE) {
-                vm_do_trace(func, fiber->stacktop - fiber->stackstart, stack);
+                vm_do_trace(func, fiber->stacktop - fiber->stackstart, fiber->data + fiber->stackstart);
             }
             janet_stack_frame(stack)->pc = pc;
             if (janet_fiber_funcframe(fiber, func)) {
@@ -961,7 +977,7 @@ static JanetSignal run_vm(JanetFiber *fiber, Janet in) {
         if (janet_checktype(callee, JANET_FUNCTION)) {
             func = janet_unwrap_function(callee);
             if (func->gc.flags & JANET_FUNCFLAG_TRACE) {
-                vm_do_trace(func, fiber->stacktop - fiber->stackstart, stack);
+                vm_do_trace(func, fiber->stacktop - fiber->stackstart, fiber->data + fiber->stackstart);
             }
             if (janet_fiber_funcframe_tail(fiber, func)) {
                 janet_stack_frame(fiber->data + fiber->frame)->pc = pc;
@@ -997,8 +1013,12 @@ static JanetSignal run_vm(JanetFiber *fiber, Janet in) {
         Janet retreg;
         vm_assert_type(stack[B], JANET_FIBER);
         JanetFiber *child = janet_unwrap_fiber(stack[B]);
+        if (janet_check_can_resume(child, &retreg)) {
+            vm_commit();
+            janet_panicv(retreg);
+        }
         fiber->child = child;
-        JanetSignal sig = janet_continue(child, stack[C], &retreg);
+        JanetSignal sig = janet_continue_no_check(child, stack[C], &retreg);
         if (sig != JANET_SIGNAL_OK && !(child->flags & (1 << sig))) {
             vm_return(sig, retreg);
         }
@@ -1234,15 +1254,14 @@ Janet janet_call(JanetFunction *fun, int32_t argc, const Janet *argv) {
     janet_vm_stackn = oldn;
     janet_gcunlock(handle);
 
-    if (signal != JANET_SIGNAL_OK) janet_panicv(*janet_vm_return_reg);
+    if (signal != JANET_SIGNAL_OK) {
+        janet_panicv(*janet_vm_return_reg);
+    }
 
     return *janet_vm_return_reg;
 }
 
-/* Enter the main vm loop */
-JanetSignal janet_continue(JanetFiber *fiber, Janet in, Janet *out) {
-    jmp_buf buf;
-
+static JanetSignal janet_check_can_resume(JanetFiber *fiber, Janet *out) {
     /* Check conditions */
     JanetFiberStatus old_status = janet_fiber_status(fiber);
     if (janet_vm_stackn >= JANET_RECURSION_GUARD) {
@@ -1259,13 +1278,22 @@ JanetSignal janet_continue(JanetFiber *fiber, Janet in, Janet *out) {
         *out = janet_wrap_string(str);
         return JANET_SIGNAL_ERROR;
     }
+    return JANET_SIGNAL_OK;
+}
+
+static JanetSignal janet_continue_no_check(JanetFiber *fiber, Janet in, Janet *out) {
+    jmp_buf buf;
+
+    JanetFiberStatus old_status = janet_fiber_status(fiber);
 
     /* Continue child fiber if it exists */
     if (fiber->child) {
+        if (janet_vm_root_fiber == NULL) janet_vm_root_fiber = fiber;
         JanetFiber *child = fiber->child;
         janet_vm_stackn++;
         JanetSignal sig = janet_continue(child, in, &in);
         janet_vm_stackn--;
+        if (janet_vm_root_fiber == fiber) janet_vm_root_fiber = NULL;
         if (sig != JANET_SIGNAL_OK && !(child->flags & (1 << sig))) {
             *out = in;
             return sig;
@@ -1294,6 +1322,7 @@ JanetSignal janet_continue(JanetFiber *fiber, Janet in, Janet *out) {
     Janet *old_vm_return_reg = janet_vm_return_reg;
 
     /* Setup fiber */
+    if (janet_vm_root_fiber == NULL) janet_vm_root_fiber = fiber;
     janet_vm_fiber = fiber;
     janet_gcroot(janet_wrap_fiber(fiber));
     janet_fiber_set_status(fiber, JANET_STATUS_ALIVE);
@@ -1319,6 +1348,7 @@ JanetSignal janet_continue(JanetFiber *fiber, Janet in, Janet *out) {
     janet_gcunroot(janet_wrap_fiber(fiber));
 
     /* Restore global state */
+    if (janet_vm_root_fiber == fiber) janet_vm_root_fiber = NULL;
     janet_vm_gc_suspend = handle;
     janet_vm_fiber = old_vm_fiber;
     janet_vm_stackn = oldn;
@@ -1328,6 +1358,14 @@ JanetSignal janet_continue(JanetFiber *fiber, Janet in, Janet *out) {
     return signal;
 }
 
+/* Enter the main vm loop */
+JanetSignal janet_continue(JanetFiber *fiber, Janet in, Janet *out) {
+    /* Check conditions */
+    JanetSignal tmp_signal = janet_check_can_resume(fiber, out);
+    if (tmp_signal) return tmp_signal;
+    return janet_continue_no_check(fiber, in, out);
+}
+
 JanetSignal janet_pcall(
     JanetFunction *fun,
     int32_t argc,
@@ -1365,11 +1403,8 @@ int janet_init(void) {
     /* Garbage collection */
     janet_vm_blocks = NULL;
     janet_vm_next_collection = 0;
-    /* Setting memoryInterval to zero forces
-     * a collection pretty much every cycle, which is
-     * incredibly horrible for performance, but can help ensure
-     * there are no memory bugs during development */
-    janet_vm_gc_interval = 0x10000;
+    janet_vm_gc_interval = 0x400000;
+    janet_vm_block_count = 0;
     janet_symcache_init();
     /* Initialize gc roots */
     janet_vm_roots = NULL;
@@ -1384,10 +1419,20 @@ int janet_init(void) {
     janet_vm_abstract_registry = janet_table(0);
     janet_gcroot(janet_wrap_table(janet_vm_registry));
     janet_gcroot(janet_wrap_table(janet_vm_abstract_registry));
+    /* Traversal */
+    janet_vm_traversal = NULL;
+    janet_vm_traversal_base = NULL;
+    janet_vm_traversal_top = NULL;
     /* Core env */
     janet_vm_core_env = NULL;
+    /* Dynamic bindings */
+    janet_vm_top_dyns = NULL;
     /* Seed RNG */
     janet_rng_seed(janet_default_rng(), 0);
+    /* Fibers */
+    janet_vm_fiber = NULL;
+    janet_vm_root_fiber = NULL;
+    janet_vm_stackn = 0;
     /* Threads */
 #ifdef JANET_THREADS
     janet_threads_init();
@@ -1406,7 +1451,14 @@ void janet_deinit(void) {
     janet_vm_registry = NULL;
     janet_vm_abstract_registry = NULL;
     janet_vm_core_env = NULL;
+    janet_vm_top_dyns = NULL;
+    free(janet_vm_traversal_base);
+    janet_vm_fiber = NULL;
+    janet_vm_root_fiber = NULL;
 #ifdef JANET_THREADS
     janet_threads_deinit();
 #endif
+#ifdef JANET_NET
+    janet_net_deinit();
+#endif
 }

src/include/janet.h

@@ -27,6 +27,12 @@
 extern "C" {
 #endif
 
+/* Variable length arrays are ok */
+#ifdef _MSC_VER
+#pragma warning( push )
+#pragma warning( disable : 4200 )
+#endif
+
 /***** START SECTION CONFIG *****/
 
 #include "janetconf.h"
@@ -121,9 +127,21 @@ extern "C" {
 #define JANET_LITTLE_ENDIAN 1
 #endif
 
+/* Limits for converting doubles to 64 bit integers */
+#define JANET_INTMAX_DOUBLE 9007199254740992.0
+#define JANET_INTMIN_DOUBLE (-9007199254740992.0)
+#define JANET_INTMAX_INT64 9007199254740992
+#define JANET_INTMIN_INT64 (-9007199254740992)
+
 /* Check emscripten */
 #ifdef __EMSCRIPTEN__
 #define JANET_NO_DYNAMIC_MODULES
+#define JANET_NO_PROCESSES
+#endif
+
+/* Check sun */
+#ifdef __sun
+#define JANET_NO_UTC_MKTIME
 #endif
 
 /* Define how global janet state is declared */
@@ -159,6 +177,11 @@ extern "C" {
 #define JANET_TYPED_ARRAY
 #endif
 
+/* Enable or disable networking */
+#if !defined(JANET_NO_NET) && !defined(__EMSCRIPTEN__)
+#define JANET_NET
+#endif
+
 /* Enable or disable large int types (for now 64 bit, maybe 128 / 256 bit integer types) */
 #ifndef JANET_NO_INT_TYPES
 #define JANET_INT_TYPES
@@ -289,6 +312,8 @@ typedef enum {
     JANET_SIGNAL_USER9
 } JanetSignal;
 
+#define JANET_SIGNAL_EVENT JANET_SIGNAL_USER9
+
 /* Fiber statuses - mostly corresponds to signals. */
 typedef enum {
     JANET_STATUS_DEAD,
@@ -309,14 +334,6 @@ typedef enum {
     JANET_STATUS_ALIVE
 } JanetFiberStatus;
 
-#ifdef JANET_NANBOX_64
-typedef union Janet Janet;
-#elif defined(JANET_NANBOX_32)
-typedef union Janet Janet;
-#else
-typedef struct Janet Janet;
-#endif
-
 /* Use type punning for GC objects */
 typedef struct JanetGCObject JanetGCObject;
 
@@ -347,15 +364,6 @@ typedef struct JanetByteView JanetByteView;
 typedef struct JanetDictView JanetDictView;
 typedef struct JanetRange JanetRange;
 typedef struct JanetRNG JanetRNG;
-typedef Janet(*JanetCFunction)(int32_t argc, Janet *argv);
-
-/* String and other aliased pointer types */
-typedef const uint8_t *JanetString;
-typedef const uint8_t *JanetSymbol;
-typedef const uint8_t *JanetKeyword;
-typedef const Janet *JanetTuple;
-typedef const JanetKV *JanetStruct;
-typedef void *JanetAbstract;
 
 /* Basic types for all Janet Values */
 typedef enum JanetType {
@@ -377,6 +385,61 @@ typedef enum JanetType {
     JANET_POINTER
 } JanetType;
 
+/* Recursive type (Janet) */
+#ifdef JANET_NANBOX_64
+typedef union Janet Janet;
+union Janet {
+    uint64_t u64;
+    int64_t i64;
+    double number;
+    void *pointer;
+};
+#elif defined(JANET_NANBOX_32)
+typedef union Janet Janet;
+union Janet {
+    struct {
+#ifdef JANET_BIG_ENDIAN
+        uint32_t type;
+        union {
+            int32_t integer;
+            void *pointer;
+        } payload;
+#else
+        union {
+            int32_t integer;
+            void *pointer;
+        } payload;
+        uint32_t type;
+#endif
+    } tagged;
+    double number;
+    uint64_t u64;
+};
+#else
+typedef struct Janet Janet;
+struct Janet {
+    union {
+        uint64_t u64;
+        double number;
+        int32_t integer;
+        void *pointer;
+        const void *cpointer;
+    } as;
+    JanetType type;
+};
+#endif
+
+/* C functions */
+typedef Janet(*JanetCFunction)(int32_t argc, Janet *argv);
+
+/* String and other aliased pointer types */
+typedef const uint8_t *JanetString;
+typedef const uint8_t *JanetSymbol;
+typedef const uint8_t *JanetKeyword;
+typedef const Janet *JanetTuple;
+typedef const JanetKV *JanetStruct;
+typedef void *JanetAbstract;
+
 #define JANET_COUNT_TYPES (JANET_POINTER + 1)
 
 /* Type flags */
@@ -485,13 +548,6 @@ JANET_API Janet janet_wrap_integer(int32_t x);
 
 #include <math.h>
 
-/* 64 Nanboxed Janet value */
-union Janet {
-    uint64_t u64;
-    int64_t i64;
-    double number;
-    void *pointer;
-};
 #define janet_u64(x) ((x).u64)
 
 #define JANET_NANBOX_TAGBITS     0xFFFF800000000000llu
@@ -576,27 +632,6 @@ JANET_API Janet janet_nanbox_from_bits(uint64_t bits);
 
 #elif defined(JANET_NANBOX_32)
 
-/* 32 bit nanboxed janet */
-union Janet {
-    struct {
-#ifdef JANET_BIG_ENDIAN
-        uint32_t type;
-        union {
-            int32_t integer;
-            void *pointer;
-        } payload;
-#else
-        union {
-            int32_t integer;
-            void *pointer;
-        } payload;
-        uint32_t type;
-#endif
-    } tagged;
-    double number;
-    uint64_t u64;
-};
-
 #define JANET_DOUBLE_OFFSET 0xFFFF
 
 #define janet_u64(x) ((x).u64)
@@ -647,18 +682,6 @@ JANET_API Janet janet_nanbox32_from_tagp(uint32_t tag, void *pointer);
 
 #else
 
-/* A general janet value type for more standard C */
-struct Janet {
-    union {
-        uint64_t u64;
-        double number;
-        int32_t integer;
-        void *pointer;
-        const void *cpointer;
-    } as;
-    JanetType type;
-};
-
 #define janet_u64(x) ((x).as.u64)
 #define janet_type(x) ((x).type)
 #define janet_checktype(x, t) ((x).type == (t))
@@ -689,7 +712,7 @@ JANET_API int janet_checkint64(Janet x);
 JANET_API int janet_checksize(Janet x);
 JANET_API JanetAbstract janet_checkabstract(Janet x, const JanetAbstractType *at);
 #define janet_checkintrange(x) ((x) >= INT32_MIN && (x) <= INT32_MAX && (x) == (int32_t)(x))
-#define janet_checkint64range(x) ((x) >= INT64_MIN && (x) <= INT64_MAX && (x) == (int64_t)(x))
+#define janet_checkint64range(x) ((x) >= JANET_INTMIN_DOUBLE && (x) <= JANET_INTMAX_DOUBLE && (x) == (int64_t)(x))
 #define janet_unwrap_integer(x) ((int32_t) janet_unwrap_number(x))
 #define janet_wrap_integer(x) janet_wrap_number((int32_t)(x))
 
@@ -733,8 +756,9 @@ struct JanetStackFrame {
     int32_t flags;
 };
 
-/* Number of Janets a frame takes up in the stack */
-#define JANET_FRAME_SIZE ((sizeof(JanetStackFrame) + sizeof(Janet) - 1) / sizeof(Janet))
+/* Number of Janets a frame takes up in the stack
+ * Should be constant across architectures */
+#define JANET_FRAME_SIZE 4
 
 /* A dynamic array type. */
 struct JanetArray {
@@ -978,7 +1002,7 @@ struct JanetRNG {
 typedef struct JanetFile JanetFile;
 struct JanetFile {
     FILE *file;
-    int flags;
+    int32_t flags;
 };
 
 /* Thread types */
@@ -1100,6 +1124,8 @@ enum JanetOpCode {
     JOP_GREATER_THAN_EQUAL,
     JOP_LESS_THAN_EQUAL,
     JOP_NEXT,
+    JOP_NOT_EQUALS,
+    JOP_NOT_EQUALS_IMMEDIATE,
     JOP_INSTRUCTION_COUNT
 };
 
@@ -1110,6 +1136,11 @@ extern enum JanetInstructionType janet_instructions[JOP_INSTRUCTION_COUNT];
 
 /***** START SECTION MAIN *****/
 
+/* Event Loop */
+#ifdef JANET_NET
+JANET_API void janet_loop(void);
+#endif
+
 /* Parsing */
 extern JANET_API const JanetAbstractType janet_parser_type;
 JANET_API void janet_parser_init(JanetParser *parser);
@@ -1208,6 +1239,7 @@ JANET_API void janet_buffer_push_u64(JanetBuffer *buffer, uint64_t x);
 #define JANET_TUPLE_FLAG_BRACKETCTOR 0x10000
 
 #define janet_tuple_head(t) ((JanetTupleHead *)((char *)t - offsetof(JanetTupleHead, data)))
+#define janet_tuple_from_head(gcobject) ((const Janet *)((char *)gcobject + offsetof(JanetTupleHead, data)))
 #define janet_tuple_length(t) (janet_tuple_head(t)->length)
 #define janet_tuple_hash(t) (janet_tuple_head(t)->hash)
 #define janet_tuple_sm_line(t) (janet_tuple_head(t)->sm_line)
@@ -1216,8 +1248,6 @@ JANET_API void janet_buffer_push_u64(JanetBuffer *buffer, uint64_t x);
 JANET_API Janet *janet_tuple_begin(int32_t length);
 JANET_API JanetTuple janet_tuple_end(Janet *tuple);
 JANET_API JanetTuple janet_tuple_n(const Janet *values, int32_t n);
-JANET_API int janet_tuple_equal(JanetTuple lhs, JanetTuple rhs);
-JANET_API int janet_tuple_compare(JanetTuple lhs, JanetTuple rhs);
 
 /* String/Symbol functions */
 #define janet_string_head(s) ((JanetStringHead *)((char *)s - offsetof(JanetStringHead, data)))
@@ -1237,7 +1267,8 @@ JANET_API void janet_description_b(JanetBuffer *buffer, Janet x);
 #define janet_cstringv(cstr) janet_wrap_string(janet_cstring(cstr))
 #define janet_stringv(str, len) janet_wrap_string(janet_string((str), (len)))
 JANET_API JanetString janet_formatc(const char *format, ...);
-JANET_API void janet_formatb(JanetBuffer *bufp, const char *format, va_list args);
+JANET_API JanetBuffer *janet_formatb(JanetBuffer *bufp, const char *format, ...);
+JANET_API void janet_formatbv(JanetBuffer *bufp, const char *format, va_list args);
 
 /* Symbol functions */
 JANET_API JanetSymbol janet_symbol(const uint8_t *str, int32_t len);
@@ -1254,6 +1285,7 @@ JANET_API JanetSymbol janet_symbol_gen(void);
 
 /* Structs */
 #define janet_struct_head(t) ((JanetStructHead *)((char *)t - offsetof(JanetStructHead, data)))
+#define janet_struct_from_head(t) ((const JanetKV *)((char *)gcobject + offsetof(JanetStructHead, data)))
 #define janet_struct_length(t) (janet_struct_head(t)->length)
 #define janet_struct_capacity(t) (janet_struct_head(t)->capacity)
 #define janet_struct_hash(t) (janet_struct_head(t)->hash)
@@ -1262,8 +1294,6 @@ JANET_API void janet_struct_put(JanetKV *st, Janet key, Janet value);
 JANET_API JanetStruct janet_struct_end(JanetKV *st);
 JANET_API Janet janet_struct_get(JanetStruct st, Janet key);
 JANET_API JanetTable *janet_struct_to_table(JanetStruct st);
-JANET_API int janet_struct_equal(JanetStruct lhs, JanetStruct rhs);
-JANET_API int janet_struct_compare(JanetStruct lhs, JanetStruct rhs);
 JANET_API const JanetKV *janet_struct_find(JanetStruct st, Janet key);
 
 /* Table functions */
@@ -1280,12 +1310,14 @@ JANET_API void janet_table_merge_table(JanetTable *table, JanetTable *other);
 JANET_API void janet_table_merge_struct(JanetTable *table, JanetStruct other);
 JANET_API JanetKV *janet_table_find(JanetTable *t, Janet key);
 JANET_API JanetTable *janet_table_clone(JanetTable *table);
+JANET_API void janet_table_clear(JanetTable *table);
 
 /* Fiber */
 JANET_API JanetFiber *janet_fiber(JanetFunction *callee, int32_t capacity, int32_t argc, const Janet *argv);
 JANET_API JanetFiber *janet_fiber_reset(JanetFiber *fiber, JanetFunction *callee, int32_t argc, const Janet *argv);
 JANET_API JanetFiberStatus janet_fiber_status(JanetFiber *fiber);
 JANET_API JanetFiber *janet_current_fiber(void);
+JANET_API JanetFiber *janet_root_fiber(void);
 
 /* Treat similar types through uniform interfaces for iteration */
 JANET_API int janet_indexed_view(Janet seq, const Janet **data, int32_t *len);
@@ -1296,6 +1328,7 @@ JANET_API const JanetKV *janet_dictionary_next(const JanetKV *kvs, int32_t cap,
 
 /* Abstract */
 #define janet_abstract_head(u) ((JanetAbstractHead *)((char *)u - offsetof(JanetAbstractHead, data)))
+#define janet_abstract_from_head(gcobject) ((JanetAbstract)((char *)gcobject + offsetof(JanetAbstractHead, data)))
 #define janet_abstract_type(u) (janet_abstract_head(u)->type)
 #define janet_abstract_size(u) (janet_abstract_head(u)->size)
 JANET_API void *janet_abstract_begin(const JanetAbstractType *type, size_t size);
@@ -1308,6 +1341,8 @@ typedef JanetBuildConfig(*JanetModconf)(void);
 JANET_API JanetModule janet_native(const char *name, JanetString *error);
 
 /* Marshaling */
+#define JANET_MARSHAL_UNSAFE 0x20000
+
 JANET_API void janet_marshal(
     JanetBuffer *buf,
     Janet x,
@@ -1342,10 +1377,11 @@ JANET_API int janet_verify(JanetFuncDef *def);
 /* Pretty printing */
 #define JANET_PRETTY_COLOR 1
 #define JANET_PRETTY_ONELINE 2
+#define JANET_PRETTY_NOTRUNC 4
 JANET_API JanetBuffer *janet_pretty(JanetBuffer *buffer, int depth, int flags, Janet x);
 
 /* Misc */
-#ifndef JANET_NO_PRF
+#ifdef JANET_PRF
 #define JANET_HASH_KEY_SIZE 16
 JANET_API void janet_init_hash_key(uint8_t key[JANET_HASH_KEY_SIZE]);
 #endif
@@ -1498,11 +1534,11 @@ extern JANET_API const JanetAbstractType janet_file_type;
 #define JANET_FILE_SERIALIZABLE 128
 #define JANET_FILE_PIPED 256
 
-JANET_API Janet janet_makefile(FILE *f, int flags);
-JANET_API FILE *janet_getfile(const Janet *argv, int32_t n, int *flags);
+JANET_API Janet janet_makefile(FILE *f, int32_t flags);
+JANET_API FILE *janet_getfile(const Janet *argv, int32_t n, int32_t *flags);
 JANET_API FILE *janet_dynfile(const char *name, FILE *def);
 JANET_API JanetAbstract janet_checkfile(Janet j);
-JANET_API FILE *janet_unwrapfile(Janet j, int *flags);
+JANET_API FILE *janet_unwrapfile(Janet j, int32_t *flags);
 
 /* Marshal API */
 JANET_API void janet_marshal_size(JanetMarshalContext *ctx, size_t value);
@@ -1555,6 +1591,9 @@ typedef enum {
     RULE_ERROR,        /* [rule] */
     RULE_DROP,         /* [rule] */
     RULE_BACKMATCH,    /* [tag] */
+    RULE_TO,           /* [rule] */
+    RULE_THRU,         /* [rule] */
+    RULE_LENPREFIX,    /* [rule_a, rule_b (repeat rule_b rule_a times)] */
 } JanetPegOpcode;
 
 typedef struct {
@@ -1651,6 +1690,11 @@ JANET_API int janet_thread_send(JanetThread *thread, Janet msg, double timeout);
 
 /***** END SECTION MAIN *****/
 
+/* Re-enable popped variable length array warnings */
+#ifdef _MSC_VER
+#pragma warning( pop )
+#endif
+
 #ifdef __cplusplus
 }
 #endif

src/mainclient/shell.c

@@ -40,6 +40,8 @@ void janet_line_deinit();
 void janet_line_get(const char *p, JanetBuffer *buffer);
 Janet janet_line_getter(int32_t argc, Janet *argv);
 
+static JANET_THREAD_LOCAL int gbl_cancel_current_repl_form = 0;
+
 /*
  * Line Editing
  */
@@ -54,7 +56,17 @@ Janet janet_line_getter(int32_t argc, Janet *argv) {
     gbl_complete_env = (argc >= 3) ? janet_gettable(argv, 2) : NULL;
     janet_line_get(str, buf);
     gbl_complete_env = NULL;
-    return janet_wrap_buffer(buf);
+
+    Janet result;
+    if (gbl_cancel_current_repl_form) {
+        gbl_cancel_current_repl_form = 0;
+
+        /* Signal that the user bailed out of the current form */
+        result = janet_ckeywordv("cancel");
+    } else {
+        result = janet_wrap_buffer(buf);
+    }
+    return result;
 }
 
 static void simpleline(JanetBuffer *buffer) {
@@ -72,7 +84,7 @@ static void simpleline(JanetBuffer *buffer) {
 }
 
 /* Windows */
-#ifdef JANET_WINDOWS
+#if defined(JANET_WINDOWS) || defined(JANET_SIMPLE_GETLINE)
 
 void janet_line_init() {
     ;
@@ -114,21 +126,28 @@ https://github.com/antirez/linenoise/blob/master/linenoise.c
 #define JANET_LINE_MAX 1024
 #define JANET_MATCH_MAX 256
 #define JANET_HISTORY_MAX 100
-static JANET_THREAD_LOCAL int gbl_israwmode = 0;
-static JANET_THREAD_LOCAL const char *gbl_prompt = "> ";
-static JANET_THREAD_LOCAL int gbl_plen = 2;
-static JANET_THREAD_LOCAL char gbl_buf[JANET_LINE_MAX];
-static JANET_THREAD_LOCAL int gbl_len = 0;
-static JANET_THREAD_LOCAL int gbl_pos = 0;
-static JANET_THREAD_LOCAL int gbl_cols = 80;
-static JANET_THREAD_LOCAL char *gbl_history[JANET_HISTORY_MAX];
-static JANET_THREAD_LOCAL int gbl_history_count = 0;
-static JANET_THREAD_LOCAL int gbl_historyi = 0;
-static JANET_THREAD_LOCAL int gbl_sigint_flag = 0;
-static JANET_THREAD_LOCAL struct termios gbl_termios_start;
-static JANET_THREAD_LOCAL JanetByteView gbl_matches[JANET_MATCH_MAX];
-static JANET_THREAD_LOCAL int gbl_match_count = 0;
-static JANET_THREAD_LOCAL int gbl_lines_below = 0;
+static int gbl_israwmode = 0;
+static const char *gbl_prompt = "> ";
+static int gbl_plen = 2;
+static char gbl_buf[JANET_LINE_MAX];
+static int gbl_len = 0;
+static int gbl_pos = 0;
+static int gbl_cols = 80;
+static char *gbl_history[JANET_HISTORY_MAX];
+static int gbl_history_count = 0;
+static int gbl_historyi = 0;
+static int gbl_sigint_flag = 0;
+static struct termios gbl_termios_start;
+static JanetByteView gbl_matches[JANET_MATCH_MAX];
+static int gbl_match_count = 0;
+static int gbl_lines_below = 0;
+
+/* Put a lock around this global state so we don't screw up
+ * the terminal in a multithreaded situation */
+#ifndef JANET_SINGLE_THREADED
+#include <pthread.h>
+static pthread_mutex_t gbl_lock = PTHREAD_MUTEX_INITIALIZER;
+#endif
 
 /* Unsupported terminal list from linenoise */
 static const char *badterms[] = {
@@ -150,6 +169,9 @@ static char *sdup(const char *s) {
 /* Ansi terminal raw mode */
 static int rawmode(void) {
     struct termios t;
+#ifndef JANET_SINGLE_THREADED
+    pthread_mutex_lock(&gbl_lock);
+#endif
     if (!isatty(STDIN_FILENO)) goto fatal;
     if (tcgetattr(STDIN_FILENO, &gbl_termios_start) == -1) goto fatal;
     t = gbl_termios_start;
@@ -163,6 +185,9 @@ static int rawmode(void) {
     return 0;
 fatal:
     errno = ENOTTY;
+#ifndef JANET_SINGLE_THREADED
+    pthread_mutex_unlock(&gbl_lock);
+#endif
     return -1;
 }
 
@@ -170,6 +195,9 @@ fatal:
 static void norawmode(void) {
     if (gbl_israwmode && tcsetattr(STDIN_FILENO, TCSAFLUSH, &gbl_termios_start) != -1)
         gbl_israwmode = 0;
+#ifndef JANET_SINGLE_THREADED
+    pthread_mutex_unlock(&gbl_lock);
+#endif
 }
 
 static int curpos(void) {
@@ -515,6 +543,147 @@ static void check_specials(JanetByteView src) {
     check_cmatch(src, "while");
 }
 
+static void resolve_format(JanetTable *entry) {
+    int is_macro = janet_truthy(janet_table_get(entry, janet_ckeywordv("macro")));
+    Janet refv = janet_table_get(entry, janet_ckeywordv("ref"));
+    int is_ref = janet_checktype(refv, JANET_ARRAY);
+    Janet value = janet_wrap_nil();
+    if (is_ref) {
+        JanetArray *a = janet_unwrap_array(refv);
+        if (a->count) value = a->data[0];
+    } else {
+        value = janet_table_get(entry, janet_ckeywordv("value"));
+    }
+    if (is_macro) {
+        fprintf(stderr, "    macro\n");
+        gbl_lines_below++;
+    } else if (is_ref) {
+        janet_eprintf("    var (%t)\n", value);
+        gbl_lines_below++;
+    } else {
+        janet_eprintf("    %t\n", value);
+        gbl_lines_below++;
+    }
+    Janet sm = janet_table_get(entry, janet_ckeywordv("source-map"));
+    Janet path = janet_get(sm, janet_wrap_integer(0));
+    Janet line = janet_get(sm, janet_wrap_integer(1));
+    Janet col = janet_get(sm, janet_wrap_integer(2));
+    if (janet_checktype(path, JANET_STRING) && janet_truthy(line) && janet_truthy(col)) {
+        janet_eprintf("    %S on line %v, column %v\n", janet_unwrap_string(path), line, col);
+        gbl_lines_below++;
+    }
+}
+
+static void doc_format(JanetString doc, int32_t width) {
+    int32_t maxcol = width - 8;
+    uint8_t wordbuf[256] = {0};
+    int32_t wordp = 0;
+    int32_t current = 0;
+    if (maxcol > 200) maxcol = 200;
+    fprintf(stderr, "    ");
+    for (int32_t i = 0; i < janet_string_length(doc); i++) {
+        uint8_t b = doc[i];
+        switch (b) {
+            default: {
+                if (maxcol <= current + wordp + 1) {
+                    if (!current) {
+                        fwrite(wordbuf, wordp, 1, stderr);
+                        wordp = 0;
+                    }
+                    fprintf(stderr, "\n    ");
+                    gbl_lines_below++;
+                    current = 0;
+                }
+                wordbuf[wordp++] = b;
+                break;
+            }
+            case '\t': {
+                if (maxcol <= current + wordp + 2) {
+                    if (!current) {
+                        fwrite(wordbuf, wordp, 1, stderr);
+                        wordp = 0;
+                    }
+                    fprintf(stderr, "\n    ");
+                    gbl_lines_below++;
+                    current = 0;
+                }
+                wordbuf[wordp++] = ' ';
+                wordbuf[wordp++] = ' ';
+                break;
+            }
+            case '\n':
+            case ' ': {
+                if (wordp) {
+                    int32_t oldcur = current;
+                    int spacer = maxcol > current + wordp + 1;
+                    if (spacer) current++;
+                    else current = 0;
+                    current += wordp;
+                    if (oldcur) fprintf(stderr, spacer ? " " : "\n    ");
+                    if (oldcur && !spacer) gbl_lines_below++;
+                    fwrite(wordbuf, wordp, 1, stderr);
+                    wordp = 0;
+                }
+                if (b == '\n') {
+                    fprintf(stderr, "\n    ");
+                    gbl_lines_below++;
+                    current = 0;
+                }
+            }
+        }
+    }
+    if (wordp) {
+        int32_t oldcur = current;
+        int spacer = maxcol > current + wordp + 1;
+        if (spacer) current++;
+        else current = 0;
+        current += wordp + 1;
+        if (oldcur) fprintf(stderr, spacer ? " " : "\n    ");
+        if (oldcur && !spacer) gbl_lines_below++;
+        fwrite(wordbuf, wordp, 1, stderr);
+        wordp = 0;
+    }
+}
+
+static void find_matches(JanetByteView prefix) {
+    JanetTable *env = gbl_complete_env;
+    gbl_match_count = 0;
+    while (NULL != env) {
+        JanetKV *kvend = env->data + env->capacity;
+        for (JanetKV *kv = env->data; kv < kvend; kv++) {
+            if (!janet_checktype(kv->key, JANET_SYMBOL)) continue;
+            const uint8_t *sym = janet_unwrap_symbol(kv->key);
+            check_match(prefix, sym, janet_string_length(sym));
+        }
+        env = env->proto;
+    }
+}
+
+static void kshowdoc(void) {
+    if (!gbl_complete_env) return;
+    while (is_symbol_char_gen(gbl_buf[gbl_pos])) gbl_pos++;
+    JanetByteView prefix = get_symprefix();
+    Janet symbol = janet_symbolv(prefix.bytes, prefix.len);
+    Janet entry = janet_table_get(gbl_complete_env, symbol);
+    if (!janet_checktype(entry, JANET_TABLE)) return;
+    Janet doc = janet_table_get(janet_unwrap_table(entry), janet_ckeywordv("doc"));
+    if (!janet_checktype(doc, JANET_STRING)) return;
+    JanetString docs = janet_unwrap_string(doc);
+    int num_cols = getcols();
+    clearlines();
+    fprintf(stderr, "\n\n\n");
+    gbl_lines_below += 3;
+    resolve_format(janet_unwrap_table(entry));
+    fprintf(stderr, "\n");
+    gbl_lines_below += 1;
+    doc_format(docs, num_cols);
+    fprintf(stderr, "\n\n");
+    gbl_lines_below += 2;
+    /* Go up to original line (zsh-like autocompletion) */
+    fprintf(stderr, "\x1B[%dA", gbl_lines_below);
+    fflush(stderr);
+}
+
 static void kshowcomp(void) {
     JanetTable *env = gbl_complete_env;
     if (env == NULL) {
@@ -528,19 +697,9 @@ static void kshowcomp(void) {
         gbl_pos++;
 
     JanetByteView prefix = get_symprefix();
-    if (prefix.len  == 0) return;
+    if (prefix.len == 0) return;
 
-    /* Find all matches */
-    gbl_match_count = 0;
-    while (NULL != env) {
-        JanetKV *kvend = env->data + env->capacity;
-        for (JanetKV *kv = env->data; kv < kvend; kv++) {
-            if (!janet_checktype(kv->key, JANET_SYMBOL)) continue;
-            const uint8_t *sym = janet_unwrap_symbol(kv->key);
-            check_match(prefix, sym, janet_string_length(sym));
-        }
-        env = env->proto;
-    }
+    find_matches(prefix);
 
     check_specials(prefix);
 
@@ -615,8 +774,7 @@ static int line() {
                 kleft();
                 break;
             case 3:     /* ctrl-c */
-                errno = EAGAIN;
-                gbl_sigint_flag = 1;
+                gbl_cancel_current_repl_form = 1;
                 clearlines();
                 return -1;
             case 4:     /* ctrl-d, eof */
@@ -633,6 +791,10 @@ static int line() {
             case 6:     /* ctrl-f */
                 kright();
                 break;
+            case 7: /* ctrl-g */
+                kshowdoc();
+                refresh();
+                break;
             case 127:   /* backspace */
             case 8:     /* ctrl-h */
                 kbackspace(1);
@@ -850,6 +1012,11 @@ int main(int argc, char **argv) {
     SetConsoleOutputCP(65001);
 #endif
 
+#if !defined(JANET_WINDOWS) && !defined(JANET_SIMPLE_GETLINE)
+    /* Try and not leave the terminal in a bad state */
+    atexit(norawmode);
+#endif
+
     /* Set up VM */
     janet_init();
 
@@ -876,10 +1043,15 @@ int main(int argc, char **argv) {
     JanetFiber *fiber = janet_fiber(janet_unwrap_function(mainfun), 64, 1, mainargs);
     fiber->env = env;
     status = janet_continue(fiber, janet_wrap_nil(), &out);
-    if (status != JANET_SIGNAL_OK) {
+    if (status != JANET_SIGNAL_OK && status != JANET_SIGNAL_EVENT) {
         janet_stacktrace(fiber, out);
     }
 
+#ifdef JANET_NET
+    status = JANET_SIGNAL_OK;
+    janet_loop();
+#endif
+
     /* Deinitialize vm */
     janet_deinit();
     janet_line_deinit();

test/fuzzers/fuzz_dostring.c

@@ -0,0 +1,45 @@
+#include <stdint.h>
+#include <string.h>
+#include <janet.h>
+
+int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
+
+    /* init Janet */
+    janet_init();
+
+    /* fuzz the parser */
+    JanetParser parser;
+    janet_parser_init(&parser);
+    for (int i = 0, done = 0; i < size; i++) {
+        switch (janet_parser_status(&parser)) {
+            case JANET_PARSE_DEAD:
+            case JANET_PARSE_ERROR:
+                done = 1;
+                break;
+            case JANET_PARSE_PENDING:
+                if (i == size) {
+                    janet_parser_eof(&parser);
+                } else {
+                    janet_parser_consume(&parser, data[i]);
+                }
+                break;
+            case JANET_PARSE_ROOT:
+                if (i >= size) {
+                    janet_parser_eof(&parser);
+                } else {
+                    janet_parser_consume(&parser, data[i]);
+                }
+                break;
+        }
+
+        if (done == 1)
+            break;
+    }
+    janet_parser_deinit(&parser);
+
+    /* cleanup Janet */
+    janet_deinit();
+
+    return 0;
+}
+

test/helper.janet

@@ -8,7 +8,8 @@
 
 (defn assert
   "Override's the default assert with some nice error handling."
-  [x e]
+  [x &opt e]
+  (default e "assert error")
   (++ num-tests-run)
   (when x (++ num-tests-passed))
   (if x

test/install/.gitignore

@@ -1,4 +1,5 @@
 /build
+/modpath
 .cache
 .manifests
 json.*

test/suite0.janet

@@ -206,6 +206,10 @@
 (def 🐮 :cow)
 (assert (= (string "🐼" 🦊 🐮) "🐼foxcow") "emojis 🙉 :)")
 (assert (not= 🦊 "🦊") "utf8 strings are not symbols and vice versa")
+(assert (= "\U01F637" "😷") "unicode escape 1")
+(assert (= "\u2623" "\U002623" "☣") "unicode escape 2")
+(assert (= "\u24c2" "\U0024c2" "Ⓜ") "unicode escape 3")
+(assert (= "\u0061" "a") "unicode escape 4")
 
 # Symbols with @ character
 
@@ -250,6 +254,11 @@
 (assert (apply <= (merge @[1 3 5] @[2 4 6 6 6 9])) "merge sort merge 3")
 (assert (apply <= (merge '(1 3 5) @[2 4 6 6 6 9])) "merge sort merge 4")
 
+(assert (deep= @[1 2 3 4 5] (sort @[5 3 4 1 2])) "sort 1")
+(assert (deep= @[{:a 1} {:a 4} {:a 7}] (sort-by |($ :a) @[{:a 4} {:a 7} {:a 1}])) "sort 2")
+(assert (deep= @[1 2 3 4 5] (sorted [5 3 4 1 2])) "sort 3")
+(assert (deep= @[{:a 1} {:a 4} {:a 7}] (sorted-by |($ :a) [{:a 4} {:a 7} {:a 1}])) "sort 4")
+
 # Gensym tests
 
 (assert (not= (gensym) (gensym)) "two gensyms not equal")
@@ -319,5 +328,94 @@
 (assert (= true ;(map truthy? [0 "" true @{} {} [] '()])) "truthy values")
 (assert (= false ;(map truthy? [nil false])) "non-truthy values")
 
+# Struct and Table duplicate elements
+(assert (= {:a 3 :b 2} {:a 1 :b 2 :a 3}) "struct literal duplicate keys")
+(assert (= {:a 3 :b 2} (struct :a 1 :b 2 :a 3)) "struct constructor duplicate keys")
+(assert (deep= @{:a 3 :b 2} @{:a 1 :b 2 :a 3}) "table literal duplicate keys")
+(assert (deep= @{:a 3 :b 2} (table :a 1 :b 2 :a 3)) "table constructor duplicate keys")
+
+## Polymorphic comparison -- Issue #272
+
+# confirm polymorphic comparison delegation to primitive comparators:
+(assert (= 0 (cmp 3 3)) "compare-primitive integers (1)")
+(assert (= -1 (cmp 3 5)) "compare-primitive integers (2)")
+(assert (= 1 (cmp "foo" "bar")) "compare-primitive strings")
+(assert (= 0 (compare 1 1)) "compare integers (1)")
+(assert (= -1 (compare 1 2)) "compare integers (2)")
+(assert (= 1 (compare "foo" "bar")) "compare strings (1)")
+
+(assert (compare< 1 2 3 4 5 6) "compare less than integers")
+(assert (not (compare> 1 2 3 4 5 6)) "compare not greater than integers")
+(assert (compare< 1.0 2.0 3.0 4.0 5.0 6.0) "compare less than reals")
+(assert (compare> 6 5 4 3 2 1) "compare greater than integers")
+(assert (compare> 6.0 5.0 4.0 3.0 2.0 1.0) "compare greater than reals")
+(assert (not (compare< 6.0 5.0 4.0 3.0 2.0 1.0)) "compare less than reals")
+(assert (compare<= 1 2 3 3 4 5 6) "compare less than or equal to integers")
+(assert (compare<= 1.0 2.0 3.0 3.0 4.0 5.0 6.0) "compare less than or equal to reals")
+(assert (compare>= 6 5 4 4 3 2 1) "compare greater than or equal to integers")
+(assert (compare>= 6.0 5.0 4.0 4.0 3.0 2.0 1.0) "compare greater than or equal to reals")
+(assert (compare< 1.0 nil false true
+           (fiber/new (fn [] 1))
+           "hi"
+           (quote hello)
+           :hello
+           (array 1 2 3)
+           (tuple 1 2 3)
+           (table "a" "b" "c" "d")
+           (struct 1 2 3 4)
+           (buffer "hi")
+           (fn [x] (+ x x))
+           print) "compare type ordering")
+
+# test polymorphic compare with 'objects' (table/setproto)
+(def mynum
+  @{:type :mynum :v 0 :compare
+    (fn [self other]
+      (case (type other)
+      :number (cmp (self :v) other)
+      :table (when (= (get other :type) :mynum)
+               (cmp (self :v) (other :v)))))})
+
+(let [n3 (table/setproto @{:v 3} mynum)]
+  (assert (= 0 (compare 3 n3)) "compare num to object (1)")
+  (assert (= -1 (compare n3 4)) "compare object to num (2)")
+  (assert (= 1 (compare (table/setproto @{:v 4} mynum) n3)) "compare object to object")
+  (assert (compare< 2 n3 4) "compare< poly")
+  (assert (compare> 4 n3 2) "compare> poly")
+  (assert (compare<= 2 3 n3 4) "compare<= poly")
+  (assert (compare= 3 n3 (table/setproto @{:v 3} mynum)) "compare= poly")
+  (assert (deep= (sorted @[4 5 n3 2] compare<) @[2 n3 4 5]) "polymorphic sort"))
+
+(let [MAX_INT_64_STRING "9223372036854775807"
+      MAX_UINT_64_STRING "18446744073709551615"
+      MAX_INT_IN_DBL_STRING "9007199254740991"
+      NAN (math/log -1)
+      INF (/ 1 0)
+      MINUS_INF (/ -1 0)
+      compare-poly-tests
+      [[(int/s64 3) (int/u64 3) 0]
+       [(int/s64 -3) (int/u64 3) -1]
+       [(int/s64 3) (int/u64 2) 1] 
+       [(int/s64 3) 3 0] [(int/s64 3) 4 -1] [(int/s64 3) -9 1]
+       [(int/u64 3) 3 0] [(int/u64 3) 4 -1] [(int/u64 3) -9 1]
+       [3 (int/s64 3) 0] [3 (int/s64 4) -1] [3 (int/s64 -5) 1]
+       [3 (int/u64 3) 0] [3 (int/u64 4) -1] [3 (int/u64 2) 1]
+       [(int/s64 MAX_INT_64_STRING) (int/u64 MAX_UINT_64_STRING) -1]
+       [(int/s64 MAX_INT_IN_DBL_STRING) (scan-number MAX_INT_IN_DBL_STRING) 0]
+       [(int/u64 MAX_INT_IN_DBL_STRING) (scan-number MAX_INT_IN_DBL_STRING) 0]
+       [(+ 1 (int/u64 MAX_INT_IN_DBL_STRING)) (scan-number MAX_INT_IN_DBL_STRING) 1]
+       [(int/s64 0) INF -1] [(int/u64 0) INF -1]
+       [MINUS_INF (int/u64 0) -1] [MINUS_INF (int/s64 0) -1] 
+       [(int/s64 1) NAN 0] [NAN (int/u64 1) 0]]]
+  (each [x y c] compare-poly-tests
+    (assert (= c (compare x y)) (string/format "compare polymorphic %q %q %d" x y c))))
+
+(assert (= nil (any? [])) "any? 1")
+(assert (= nil (any? [false nil])) "any? 2")
+(assert (= nil (any? [nil false])) "any? 3")
+(assert (= 1 (any? [1])) "any? 4")
+(assert (nan? (any? [nil math/nan nil])) "any? 5")
+(assert (= true (any? [nil nil false nil nil true nil nil nil nil false :a nil])) "any? 6")
+
 (end-suite)
 

test/suite3.janet

@@ -91,8 +91,8 @@
 # Assembly test
 # Fibonacci sequence, implemented with naive recursion.
 (def fibasm (asm '{
-  arity 1
-  bytecode [
+  :arity 1
+  :bytecode [
     (ltim 1 0 0x2)      # $1 = $0 < 2
     (jmpif 1 :done)     # if ($1) goto :done
     (lds 1)             # $1 = self

test/suite4.janet

@@ -48,7 +48,9 @@
 (defn check-image
   "Run a marshaling test using the make-image and load-image functions."
   [x msg]
-  (assert-no-error msg (load-image (make-image x))))
+  (def im (make-image x))
+  # (printf "\nimage-hash: %d" (-> im string hash))
+  (assert-no-error msg (load-image im)))
 
 (check-image (fn [] (fn [] 1)) "marshal nested functions")
 (check-image (fiber/new (fn [] (fn [] 1))) "marshal nested functions in fiber")

test/suite5.janet

@@ -58,6 +58,17 @@
 
 (assert (= ((unmarshal (marshal b)) 3) (b 3)) "marshal")
 
+# Issue 408
+(assert-error :invalid-type (tarray/new :int32 10 1 0 (int/u64 7)) "tarray/new should only allow tarray or buffer for last argument")
+(def ta (tarray/new :int32 10))
+(assert (= (next a nil) 0) "tarray next 1")
+(assert (= (next a 0) 1) "tarray next 2")
+(assert (= (next a 8) 9) "tarray next 3")
+(assert (nil? (next a 9)) "tarray next 4")
+(put ta 3 7)
+(put ta 9 7)
+(assert (= 2 (count |(= $ 7) ta)) "tarray count")
+
 # Array remove
 
 (assert (deep= (array/remove @[1 2 3 4 5] 2) @[1 2 4 5]) "array/remove 1")

test/suite7.janet

@@ -23,7 +23,6 @@
 
 # Using a large test grammar
 
-(def- core-env (table/getproto (fiber/getenv (fiber/current))))
 (def- specials {'fn true
                'var true
                'do true
@@ -41,7 +40,7 @@
 (defn capture-sym
   [text]
   (def sym (symbol text))
-  [(if (or (core-env sym) (specials sym)) :coresym :symbol) text])
+  [(if (or (root-env sym) (specials sym)) :coresym :symbol) text])
 
 (def grammar
   ~{:ws (set " \v\t\r\f\n\0")
@@ -316,4 +315,9 @@
 
 (assert (= 40 counter) "if-with 1")
 
+(def a @[])
+(eachk x [:a :b :c :d]
+  (array/push a x))
+(assert (deep= (range 4) a) "eachk 1")
+
 (end-suite)

test/suite8.janet

@@ -36,7 +36,7 @@
       :loop (/ (* "[" :main "]") ,(fn [& captures]
                                     ~(while (not= (get DATA POS) 0)
                                        ,;captures)))
-      :main (any (+ :s :loop :+ :- :> :< :.)) }))
+      :main (any (+ :s :loop :+ :- :> :< :.))}))
 
 (defn bf
   "Run brainfuck."
@@ -194,4 +194,159 @@
 (assert (deep= @[] (accumulate2 + [])) "accumulate2 2")
 (assert (deep= @[] (accumulate 0 + [])) "accumulate 2")
 
+# Perm strings
+
+(assert (= (os/perm-int "rwxrwxrwx") 8r777) "perm 1")
+(assert (= (os/perm-int "rwxr-xr-x") 8r755) "perm 2")
+(assert (= (os/perm-int "rw-r--r--") 8r644) "perm 3")
+
+(assert (= (band (os/perm-int "rwxrwxrwx") 8r077) 8r077) "perm 4")
+(assert (= (band (os/perm-int "rwxr-xr-x") 8r077) 8r055) "perm 5")
+(assert (= (band (os/perm-int "rw-r--r--") 8r077) 8r044) "perm 6")
+
+(assert (= (os/perm-string 8r777) "rwxrwxrwx") "perm 7")
+(assert (= (os/perm-string 8r755) "rwxr-xr-x") "perm 8")
+(assert (= (os/perm-string 8r644) "rw-r--r--") "perm 9")
+
+# Issue #336 cases - don't segfault
+
+(assert-error "unmarshal errors 1" (unmarshal @"\xd6\xb9\xb9"))
+(assert-error "unmarshal errors 2" (unmarshal @"\xd7bc"))
+(assert-error "unmarshal errors 3" (unmarshal "\xd3\x01\xd9\x01\x62\xcf\x03\x78\x79\x7a" load-image-dict))
+(assert-error "unmarshal errors 4"
+              (unmarshal
+                @"\xD7\xCD\0e/p\x98\0\0\x03\x01\x01\x01\x02\0\0\x04\0\xCEe/p../tools
+\0\0\0/afl\0\0\x01\0erate\xDE\xDE\xDE\xDE\xDE\xDE\xDE\xDE\xDE\xDE
+\xA8\xDE\xDE\xDE\xDE\xDE\xDE\0\0\0\xDE\xDE_unmarshal_testcase3.ja
+neldb\0\0\0\xD8\x05printG\x01\0\xDE\xDE\xDE'\x03\0marshal_tes/\x02
+\0\0\0\0\0*\xFE\x01\04\x02\0\0'\x03\0\r\0\r\0\r\0\r" load-image-dict))
+
+# No segfault, valgrind clean.
+
+(def x @"\xCC\xCD.nd\x80\0\r\x1C\xCDg!\0\x07\xCC\xCD\r\x1Ce\x10\0\r;\xCDb\x04\xFF9\xFF\x80\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04uu\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\0\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04}\x04\x04\x04\x04\x04\x04\x04\x04#\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\0\x01\0\0\x03\x04\x04\x04\xE2\x03\x04\x04\x04\x04\x04\x04\x04\x04\x04\x14\x1A\x04\x04\x04\x04\x04\x18\x04\x04!\x04\xE2\x03\x04\x04\x04\x04\x04\x04$\x04\x04\x04\x04\x04\x04\x04\x04\x04\x80\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04\x04A\0\0\0\x03\0\0!\xBF\xFF")
+(unmarshal x load-image-dict)
+(gccollect)
+(marshal x make-image-dict)
+
+(def b @"\xCC\xCD\0\x03\0\x08\x04\rm\xCD\x7F\xFF\xFF\xFF\x02\0\x02\xD7\xCD\0\x98\0\0\x05\x01\x01\x01\x01\x08\xCE\x01f\xCE../tools/afl/generate_unmarshal_testcases.janet\xCE\x012,\x01\0\0&\x03\0\06\x02\x03\x03)\x03\x01\0*\x04\0\00\x03\x04\0>\x03\0\0\x03\x03\0\0*\x05\0\x11\0\x11\0\x05\0\x05\0\x05\0\x05\0\x05\xC9\xDA\x04\xC9\xC9\xC9")
+(unmarshal b load-image-dict)
+(gccollect)
+
+(def v (unmarshal
+         @"\xD7\xCD0\xD4000000\0\x03\x01\xCE\00\0\x01\0\0000\x03\0\0\0000000000\xCC0\0000"
+         load-image-dict))
+(gccollect)
+
+# in vs get regression
+(assert (nil? (first @"")) "in vs get 1")
+(assert (nil? (last @"")) "in vs get 1")
+
+# For undefined behavior sanitizer
+0xf&1fffFFFF
+
+# Tuple comparison
+(assert (< [1 2 3] [2 2 3]) "tuple comparison 1")
+(assert (< [1 2 3] [2 2]) "tuple comparison 2")
+(assert (< [1 2 3] [2 2 3 4]) "tuple comparison 3")
+(assert (< [1 2 3] [1 2 3 4]) "tuple comparison 4")
+(assert (< [1 2 3] [1 2 3 -1]) "tuple comparison 5")
+(assert (> [1 2 3] [1 2]) "tuple comparison 6")
+
+# Lenprefix rule
+
+(def peg (peg/compile ~(* (lenprefix (/ (* '(any (if-not ":" 1)) ":") ,scan-number) 1) -1)))
+
+(assert (peg/match peg "5:abcde") "lenprefix 1")
+(assert (not (peg/match peg "5:abcdef")) "lenprefix 2")
+(assert (not (peg/match peg "5:abcd")) "lenprefix 3")
+
+# Packet capture
+
+(def peg2
+  (peg/compile
+    ~{# capture packet length in tag :header-len
+      :packet-header (* (/ ':d+ ,scan-number :header-len) ":")
+
+      # capture n bytes from a backref :header-len
+      :packet-body '(lenprefix (-> :header-len) 1)
+
+      # header, followed by body, and drop the :header-len capture
+      :packet (/ (* :packet-header :packet-body) ,|$1)
+
+      # any exact seqence of packets (no extra characters)
+      :main (* (any :packet) -1)}))
+
+(assert (deep= @["a" "bb" "ccc"] (peg/match peg2 "1:a2:bb3:ccc")) "lenprefix 4")
+(assert (deep= @["a" "bb" "cccccc"] (peg/match peg2 "1:a2:bb6:cccccc")) "lenprefix 5")
+(assert (= nil (peg/match peg2 "1:a2:bb:5:cccccc")) "lenprefix 6")
+(assert (= nil (peg/match peg2 "1:a2:bb:7:cccccc")) "lenprefix 7")
+
+# Regression #400
+(assert (= nil (while (and false false) (fn []) (error "should not happen"))) "strangeloop 1")
+(assert (= nil (while (not= nil nil) (fn []) (error "should not happen"))) "strangeloop 2")
+
+# Issue #412
+(assert (peg/match '(* "a" (> -1 "a") "b") "abc") "lookhead does not move cursor")
+
+(def peg3
+  ~{:main (* "(" (thru ")"))})
+
+(def peg4 (peg/compile ~(* (thru "(") '(to ")"))))
+
+(assert (peg/match peg3 "(12345)") "peg thru 1")
+(assert (not (peg/match peg3 " (12345)")) "peg thru 2")
+(assert (not (peg/match peg3 "(12345")) "peg thru 3")
+
+(assert (= "abc" (0 (peg/match peg4 "123(abc)"))) "peg thru/to 1")
+(assert (= "abc" (0 (peg/match peg4 "(abc)"))) "peg thru/to 2")
+(assert (not (peg/match peg4 "123(abc")) "peg thru/to 3")
+
+(def peg5 (peg/compile [3 "abc"]))
+
+(assert (:match peg5 "abcabcabc") "repeat alias 1")
+(assert (:match peg5 "abcabcabcac") "repeat alias 2")
+(assert (not (:match peg5 "abcabc")) "repeat alias 3")
+
+(defn check-jdn [x]
+  (assert (deep= (parse (string/format "%j" x)) x) "round trip jdn"))
+
+(check-jdn 0)
+(check-jdn nil)
+(check-jdn [])
+(check-jdn @[[] [] 1231 9.123123 -123123 0.1231231230001])
+(check-jdn -0.123123123123)
+(check-jdn 12837192371923)
+(check-jdn "a string")
+(check-jdn @"a buffer")
+
+# Issue 428
+(var result nil)
+(defn f [] (yield {:a :ok}))
+(assert-no-error "issue 428 1" (loop [{:a x} :generate (fiber/new f)] (set result x)))
+(assert (= result :ok) "issue 428 2")
+
+# Inline 3 argument get
+(assert (= 10 (do (var a 10) (set a (get '{} :a a)))) "inline get 1")
+
+# Keyword and Symbol slice
+(assert (= :keyword (keyword/slice "some_keyword_slice" 5 12)) "keyword slice")
+(assert (= 'symbol (symbol/slice "some_symbol_slice" 5 11)) "symbol slice")
+
+# Peg find and find-all
+(def p "/usr/local/bin/janet")
+(assert (= (peg/find '"n/" p) 13) "peg find 1")
+(assert (not (peg/find '"t/" p)) "peg find 2")
+(assert (deep= (peg/find-all '"/" p) @[0 4 10 14]) "peg find-all")
+
+# Peg replace and replace-all
+(var ti 0)
+(defn check-replacer
+  [x y z]
+  (assert (= (string/replace x y z) (string (peg/replace x y z))) "replacer test replace")
+  (assert (= (string/replace-all x y z) (string (peg/replace-all x y z))) "replacer test replace-all"))
+(check-replacer "abc" "Z" "abcabcabcabasciabsabc")
+(check-replacer "abc" "Z" "")
+(check-replacer "aba" "ZZZZZZ" "ababababababa")
+(check-replacer "aba" "" "ababababababa")
+
 (end-suite)

test/suite9.janet

@@ -0,0 +1,51 @@
+# Copyright (c) 2020 Calvin Rose & contributors
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+
+(import ./helper :prefix "" :exit true)
+(start-suite 9)
+
+# Net testing
+
+(defn handler
+  "Simple handler for connections."
+  [stream]
+  (defer (:close stream)
+    (def id (gensym))
+    (def b @"")
+    (:read stream 1024 b)
+    (:write stream b)
+    (buffer/clear b)))
+
+(def s (net/server "127.0.0.1" "8000" handler))
+(assert s "made server 1")
+
+(defn test-echo [msg]
+  (with [conn (net/connect "127.0.0.1" "8000")]
+    (:write conn msg)
+    (def res (:read conn 1024))
+    (assert (= (string res) msg) (string "echo " msg))))
+
+(test-echo "hello")
+(test-echo "world")
+(test-echo (string/repeat "abcd" 200))
+
+(:close s)
+
+(end-suite)

tools/afl/README.md

@@ -3,12 +3,26 @@
 To use these, you need to install afl (of course), and xterm. A tiling window manager helps manage
 many concurrent fuzzer instances.
 
+Note, afl sometimes requires system configuration, if you find AFL quitting prematurely, try manually
+launching it and addressing any error messages.
+
 ## Fuzz the parser
 ```
 $ sh ./tools/afl/prepare_to_fuzz.sh
-export NFUZZ=1
+$ export NFUZZ=1
 $ sh ./tools/afl/fuzz.sh parser
 Ctrl+C when done to close all fuzzer terminals.
 $ sh ./tools/afl/aggregate_cases.sh parser
 $ ls ./fuzz_out/parser_aggregated/
-```
+```
+
+## Fuzz the unmarshaller
+```
+$ janet ./tools/afl/generate_unmarshal_testcases.janet
+$ sh ./tools/afl/prepare_to_fuzz.sh
+$ export NFUZZ=1
+$ sh ./tools/afl/fuzz.sh unmarshal
+Ctrl+C when done to close all fuzzer terminals.
+$ sh ./tools/afl/aggregate_cases.sh unmarshal
+$ ls ./fuzz_out/unmarshal_aggregated/
+```

tools/afl/generate_unmarshal_testcases.janet

@@ -0,0 +1,49 @@
+
+(os/mkdir "./tools/afl/unmarshal_testcases/")
+
+(defn spit-case [n v]
+  (spit
+    (string "./tools/afl/unmarshal_testcases/" (string n))
+    (marshal v make-image-dict)))
+
+(def cases [
+  nil
+
+  "abc"
+
+  :def
+
+  'hij
+
+  123
+
+  (int/s64 123)
+
+  "7"
+
+  [1 2 3]
+
+  @[1 2 3]
+
+  {:a 123}
+
+  @{:b 'xyz}
+
+  (peg/compile
+    '{:a (* "a" :b "a")
+      :b (* "b" (+ :a 0) "b")
+      :main (* "(" :b ")")})
+
+  (fn f [a] (fn [] {:ab a}))
+
+  (fn f [a] (print "hello world!"))
+
+  (do
+    (defn f [a] (yield) @[1 "2"])
+    (def fb (fiber/new f))
+    (resume fb)
+    fb)
+])
+
+(eachk i cases
+  (spit-case i (in cases i)))

tools/afl/unmarshal_runner.janet

@@ -0,0 +1,6 @@
+# Unmarshal garbage.
+(def v (unmarshal (slurp ((dyn :args) 1)) load-image-dict))
+# Trigger leaks or use after free.
+(gccollect)
+# Attempt to use generated value.
+(marshal v make-image-dict)

tools/msi/janet.wxs

@@ -0,0 +1,200 @@
+<?xml version="1.0" encoding="UTF-8"?>
+
+<?define Name = "Janet" ?>
+<?define Description = "The Janet Programming Language" ?>
+<?define Manufacturer = "Janet-Lang.org" ?>
+<?define WebPage = "https://janet-lang.org" ?>
+<?ifdef env.JANET_VERSION ?>
+    <?define Version = "$(env.JANET_VERSION)" ?>
+<?else?>
+    <?define Version = "0.0.0" ?>
+<?endif?>
+<?if $(sys.BUILDARCH)="x64" ?>
+    <?define UpgradeCode="712CACD6-09AA-430A-831C-80FDFFE3F9ED" ?>
+    <?define ProgramFilesFolder="ProgramFiles64Folder" ?>
+    <?define Win64="yes" ?>
+    <?define Arch="(x64)" ?>
+<?elseif $(sys.BUILDARCH)="x86" ?>
+    <?define UpgradeCode="0411837a-82c4-4dc7-872b-134d0c1b0228" ?>
+    <?define ProgramFilesFolder="ProgramFilesFolder" ?>
+    <?define Win64="no" ?>
+    <?define Arch="(x86)" ?>
+<?else ?>
+    <?error Unsupported value of sys.BUILDARCH=$(sys.BUILDARCH)?>
+<?endif?>
+<?define BaseRegKey="Software\Microsoft\$(var.Manufacturer)\$(var.Name)" ?>
+
+
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Product Id="*"
+			 Name="$(var.Name)"
+			 Language="1033"
+			 Version="$(var.Version)"
+			 Manufacturer="$(var.Manufacturer)"
+			 UpgradeCode="$(var.UpgradeCode)">
+        <Package Compressed="yes"
+				 InstallScope="perUser"
+				 Manufacturer="$(var.Manufacturer)"
+				 Description="$(var.Description)" />
+        <MajorUpgrade DowngradeErrorMessage="A later version of [ProductName] is already installed. Setup will now exit."/>
+        <MediaTemplate EmbedCab="yes" />
+        <Property Id="DISABLEADVTSHORTCUTS" Value="1" />
+
+        <!-- Set UI images (use the -b option to light.exe to set where these files are) -->
+        <WixVariable Id="WixUIBannerBmp" Value="JanetTopBanner.png" />
+        <WixVariable Id="WixUIDialogBmp" Value="JanetDialog.png" />
+        <WixVariable Id="WixUILicenseRtf" Value="LICENSE.rtf" />
+
+        <Icon Id="Janet.ico" SourceFile="assets\icon.ico" />
+
+        <!-- Add some details to Add/Remove Programs entry -->
+        <Property Id="ARPPRODUCTICON" Value="Janet.ico" />
+        <Property Id='ARPCOMMENTS'>$(var.Description)</Property>
+        <Property Id='ARPURLINFOABOUT'>$(var.WebPage)</Property>
+        <Property Id='COMPANY'>$(var.Manufacturer)</Property>
+
+        <!-- Default to per-user installs -->
+        <Property Id="WixAppFolder" Value="WixPerUserFolder" />
+
+        <Property Id="ApplicationFolderName" Value="$(var.Name)" />
+
+        <!-- Fix WixUI_Advanced to work with x64 -->
+        <CustomAction Id="WixSetDefaultPerMachineFolderPerArch"
+            Property="WixPerMachineFolder"
+            Value="[$(var.ProgramFilesFolder)][ApplicationFolderName]"
+            Execute="immediate"/>
+        <InstallExecuteSequence>
+            <Custom Action="WixSetDefaultPerMachineFolderPerArch" Before="WixSetPerMachineFolder" />
+        </InstallExecuteSequence>
+        <InstallUISequence>
+            <Custom Action="WixSetDefaultPerMachineFolderPerArch" Before="WixSetPerMachineFolder" />
+        </InstallUISequence>
+
+        <Directory Id="TARGETDIR" Name="SourceDir">
+            <Directory Id="$(var.ProgramFilesFolder)">
+                <Directory Id="APPLICATIONFOLDER" Name="$(var.Name)">
+                    <Directory Id="BinDir" Name="bin"/>
+                    <Directory Id="CDir" Name="C"/>
+                    <Directory Id="DocsDir" Name="docs"/>
+                    <Directory Id="LibraryDir" Name="Library"/>
+                </Directory>
+            </Directory>
+            <Directory Id="ProgramMenuFolder">
+                <Directory Id="ApplicationProgramsFolder" Name="$(var.Name)" />
+            </Directory>
+        </Directory>
+
+        <!-- 
+          Define the files to be installed.
+          File/@Source is relative to where this file is compiled, the root of the repository in this case.
+          File/@Name is the destination file name, if not set it defaults to the file name part of Source.
+          Component/@Directory is the Id of the destination directory - where the directory name and
+            hierarchy is set in the section above
+        -->
+        <ComponentGroup Id="Files">
+            <Component Directory="APPLICATIONFOLDER">
+                <File Source="README.md"/>
+                <RemoveFolder Id="RemoveRootDir" On="uninstall" />
+            </Component>
+            <Component Directory="APPLICATIONFOLDER">
+                <File Source="LICENSE"/>
+            </Component>
+            <Component Directory="APPLICATIONFOLDER">
+                <File Source="assets\icon.ico"/>
+            </Component>
+
+            <Component Directory="BinDir">
+                <File Source="dist\janet.exe" KeyPath="yes">
+                    <Shortcut Id="JanetExeShortcut"
+                        Directory="ApplicationProgramsFolder"
+                        Name="$(var.Name)"
+                        Description="$(var.Description)"
+                        Icon="Janet.ico"
+                        Advertise="yes"
+                        WorkingDirectory="INSTALLFOLDER" />
+                </File>
+                <RemoveFolder Id="RemoveBinDir" On="uninstall" />
+            </Component>
+            <Component Directory="BinDir">
+                <File Source="jpm" Name="jpm.janet"/>
+            </Component>
+            <Component Directory="BinDir">
+                <File Source="tools\jpm.bat"/>
+            </Component>
+
+            <Component Directory="CDir">
+                <File Source="dist\janet.h"/>
+                <RemoveFolder Id="RemoveCDir" On="uninstall" />
+            </Component>
+            <Component Directory="CDir">
+                <File Source="dist\janetconf.h"/>
+            </Component>
+            <Component Directory="CDir">
+                <File Source="dist\janet.lib"/>
+            </Component>
+            <Component Directory="CDir">
+                <File Source="dist\janet.exp"/>
+            </Component>
+            <Component Directory="CDir">
+                <File Source="dist\janet.c"/>
+            </Component>
+            <Component Directory="CDir">
+                <File Source="dist\libjanet.lib"/>
+            </Component>
+
+            <Component Id="LibraryComponent" Directory="LibraryDir" Guid="3860e981-5f94-4002-b5d5-2d9ec0d2792d" KeyPath="yes">
+                <RemoveFolder Id="RemoveLibraryDir" On="uninstall" />
+            </Component>
+
+            <Component Id="DocsComponent" Directory="DocsDir">
+                <File Source="dist\doc.html" Name="docs.html" KeyPath="yes">
+                    <Shortcut Id="JanetDocsShortcut"
+                        Directory="ApplicationProgramsFolder"
+                        Name="$(var.Name) Documentation"
+                        Description="$(var.Description)"
+                        Advertise="yes"/>
+                </File>
+                <RemoveFolder Id="RemoveDocsDir" On="uninstall" />
+            </Component>
+        </ComponentGroup>
+
+        <Component Id="StartMenu" Directory="ApplicationProgramsFolder">
+            <RegistryValue Root="HKMU" Key="$(var.BaseRegKey)" Name="installed" Type="integer" Value="1" KeyPath="yes" />
+            <RemoveFolder Id="RemoveApplicationProgramsFolder" On="uninstall" />
+        </Component>
+
+        <!-- This component is duplicated with different conditions so that we can set system or user environment variables -->
+        <Component Id="SetEnvVarsPerMachine" Directory="ApplicationProgramsFolder" Guid="57b1e1ef-89c8-4ce4-9f0f-37618677c5a4" KeyPath="yes">
+            <Condition>ALLUSERS=1</Condition>
+            <Environment Id="PATH_PERMACHINE" Name="PATH" Value="[BinDir]" Action="set" Permanent="no" System="yes" Part="last"/>
+            <Environment Id="JANET_BINPATH_PERMACHINE" Name="JANET_BINPATH" Value="[BinDir]" Action="set" Permanent="no" System="yes"/>
+            <Environment Id="JANET_PATH_PERMACHINE" Name="JANET_PATH" Value="[LibraryDir]" Action="set" Permanent="no" System="yes" />
+            <Environment Id="JANET_HEADERPATH_PERMACHINE" Name="JANET_HEADERPATH" Value="[CDir]" Action="set" Permanent="no" System="yes"/>
+            <Environment Id="JANET_LIBPATH_PERMACHINE" Name="JANET_LIBPATH" Value="[CDir]" Action="set" Permanent="no" System="yes"/>
+        </Component>
+        <Component Id="SetEnvVarsPerUser" Directory="ApplicationProgramsFolder" Guid="128be307-488b-49aa-971a-d2ae00a1a584" KeyPath="yes">
+            <Condition>NOT ALLUSERS=1</Condition>
+            <Environment Id="PATH_PERUSER" Name="PATH" Value="[BinDir]" Action="set" Permanent="no" System="no" Part="last"/>
+            <Environment Id="JANET_BINPATH_PERUSER" Name="JANET_BINPATH" Value="[BinDir]" Action="set" Permanent="no" System="no"/>
+            <Environment Id="JANET_PATH_PERUSER" Name="JANET_PATH" Value="[LibraryDir]" Action="set" Permanent="no" System="no" />
+            <Environment Id="JANET_HEADERPATH_PERUSER" Name="JANET_HEADERPATH" Value="[CDir]" Action="set" Permanent="no" System="no"/>
+            <Environment Id="JANET_LIBPATH_PERUSER" Name="JANET_LIBPATH" Value="[CDir]" Action="set" Permanent="no" System="no"/>
+        </Component>
+
+        <Feature Id="MainFeature" Title="$(var.Name) $(var.Version)" 
+            Level="1" Absent="disallow" AllowAdvertise="no" InstallDefault="local"
+            Description="$(var.Description)">
+            <ComponentGroupRef Id="Files" />
+            <ComponentRef Id="StartMenu" />
+            <ComponentRef Id="SetEnvVarsPerMachine" />
+            <ComponentRef Id="SetEnvVarsPerUser" />
+        </Feature>
+
+        <UI>
+            <UIRef Id="WixUI_Advanced"/>
+            <!-- FindRelatedProducts runs before the user select the install scope, so we ask it to run again if the have changed the scope
+             -->
+            <Publish Dialog="InstallScopeDlg" Control="Next" Order="8" Event="DoAction" Value="FindRelatedProducts">WixAppFolder = "WixPerMachineFolder"</Publish>
+        </UI>
+    </Product>
+</Wix>

tools/patch-jpm.janet

@@ -0,0 +1,33 @@
+# Patch jpm to have the correct paths for the current install.
+# usage: janet patch-jpm.janet output --libdir=/usr/local/lib/x64-linux/ --binpath
+
+(def- argpeg
+  (peg/compile
+    '(* "--" '(to "=") "=" '(any 1))))
+
+(def- args (tuple/slice (dyn :args) 3))
+(def- len (length args))
+(var i :private 0)
+
+(def install-paths @{})
+
+# Get flags
+(each a args
+  (if-let [m (peg/match argpeg a)]
+    (let [[key value] m]
+      (put install-paths (keyword key) value))))
+
+(def- replace-peg
+  (peg/compile
+    ~(% (* '(to "###START###")
+           (constant ,(string/format "# Inserted by tools/patch-jpm.janet\n(defn- install-paths [] %j)" install-paths))
+           (thru "###END###")
+           '(any 1)))))
+
+(def source (slurp ((dyn :args) 1)))
+(def newsource (0 (peg/match replace-peg source)))
+
+(spit ((dyn :args) 2) newsource)
+
+(unless (= :windows (os/which))
+  (os/shell (string `chmod +x "` ((dyn :args) 2) `"`)))

tools/tm_lang_gen.janet

@@ -230,53 +230,53 @@
       <key>name</key>
       <string>punctuation.other.janet</string>
     </dict>
-    <!-- string>(?&lt;![\.:\w_\-=!@\$%^&amp;?|\\/&lt;&gt;*]) token match here (?![\.:\w_\-=!@\$%^&amp;?|\\/&lt;&gt;*])</string -->
+    <!-- string>(?&lt;![\.:\w_\-=!@\$%^&amp;?/&lt;&gt;*]) token match here (?![\.:\w_\-=!@\$%^&amp;?/&lt;&gt;*])</string -->
     <key>literal</key>
     <dict>
       <key>match</key>
-      <string>(?&lt;![\.:\w_\-=!@\$%^&amp;?|\\/&lt;&gt;*])(true|false|nil)(?![\.:\w_\-=!@\$%^&amp;?|\\/&lt;&gt;*])</string>
+      <string>(?&lt;![\.:\w_\-=!@\$%^&amp;?/&lt;&gt;*])(true|false|nil)(?![\.:\w_\-=!@\$%^&amp;?/&lt;&gt;*])</string>
       <key>name</key>
       <string>constant.language.janet</string>
     </dict>
     <key>corelib</key>
     <dict>
       <key>match</key>
-            <string>(?&lt;![\.:\w_\-=!@\$%^&amp;?|\\/&lt;&gt;*])(%ALLSYMBOLS%)(?![\.:\w_\-=!@\$%^&amp;?|\\/&lt;&gt;*])</string>
+            <string>(?&lt;![\.:\w_\-=!@\$%^&amp;?/&lt;&gt;*])(%ALLSYMBOLS%)(?![\.:\w_\-=!@\$%^&amp;?/&lt;&gt;*])</string>
       <key>name</key>
       <string>keyword.control.janet</string>
     </dict>
     <key>keysym</key>
     <dict>
       <key>match</key>
-      <string>(?&lt;![\.:\w_\-=!@\$%^&amp;?|\\/&lt;&gt;*]):[\.:\w_\-=!@\$%^&amp;?|\\/&lt;&gt;*]*</string>
+      <string>(?&lt;![\.:\w_\-=!@\$%^&amp;?/&lt;&gt;*]):[\.:\w_\-=!@\$%^&amp;?/&lt;&gt;*]*</string>
       <key>name</key>
       <string>constant.keyword.janet</string>
     </dict>
     <key>symbol</key>
     <dict>
       <key>match</key>
-      <string>(?&lt;![\.:\w_\-=!@\$%^&amp;?|\\/&lt;&gt;*])[\.a-zA-Z_\-=!@\$%^&amp;?|\\/&lt;&gt;*][\.:\w_\-=!@\$%^&amp;?|\\/&lt;&gt;*]*</string>
+      <string>(?&lt;![\.:\w_\-=!@\$%^&amp;?/&lt;&gt;*])[\.a-zA-Z_\-=!@\$%^&amp;?/&lt;&gt;*][\.:\w_\-=!@\$%^&amp;?/&lt;&gt;*]*</string>
       <key>name</key>
       <string>variable.other.janet</string>
     </dict>
     <key>hex-number</key>
     <dict>
       <key>match</key>
-      <string>(?&lt;![\.:\w_\-=!@\$%^&amp;?|\\/&lt;&gt;*])[-+]?0x([_\da-fA-F]+|[_\da-fA-F]+\.[_\da-fA-F]*|\.[_\da-fA-F]+)(&amp;[+-]?[\da-fA-F]+)?(?![\.:\w_\-=!@\$%^&amp;?|\\/&lt;&gt;*])</string>
+      <string>(?&lt;![\.:\w_\-=!@\$%^&amp;?/&lt;&gt;*])[-+]?0x([_\da-fA-F]+|[_\da-fA-F]+\.[_\da-fA-F]*|\.[_\da-fA-F]+)(&amp;[+-]?[\da-fA-F]+)?(?![\.:\w_\-=!@\$%^&amp;?/&lt;&gt;*])</string>
       <key>name</key>
       <string>constant.numeric.hex.janet</string>
     </dict>
     <key>dec-number</key>
     <dict>
       <key>match</key>
-      <string>(?&lt;![\.:\w_\-=!@\$%^&amp;?|\\/&lt;&gt;*])[-+]?([_\d]+|[_\d]+\.[_\d]*|\.[_\d]+)([eE&amp;][+-]?[\d]+)?(?![\.:\w_\-=!@\$%^&amp;?|\\/&lt;&gt;*])</string>
+      <string>(?&lt;![\.:\w_\-=!@\$%^&amp;?/&lt;&gt;*])[-+]?([_\d]+|[_\d]+\.[_\d]*|\.[_\d]+)([eE&amp;][+-]?[\d]+)?(?![\.:\w_\-=!@\$%^&amp;?/&lt;&gt;*])</string>
       <key>name</key>
       <string>constant.numeric.decimal.janet</string>
     </dict>
     <key>r-number</key>
     <dict>
       <key>match</key>
-      <string>(?&lt;![\.:\w_\-=!@\$%^&amp;?|\\/&lt;&gt;*])[-+]?\d\d?r([_\w]+|[_\w]+\.[_\w]*|\.[_\w]+)(&amp;[+-]?[\w]+)?(?![\.:\w_\-=!@\$%^&amp;?|\\/&lt;&gt;*])</string>
+      <string>(?&lt;![\.:\w_\-=!@\$%^&amp;?/&lt;&gt;*])[-+]?\d\d?r([_\w]+|[_\w]+\.[_\w]*|\.[_\w]+)(&amp;[+-]?[\w]+)?(?![\.:\w_\-=!@\$%^&amp;?/&lt;&gt;*])</string>
       <key>name</key>
       <string>constant.numeric.decimal.janet</string>
     </dict>
@@ -308,7 +308,7 @@
       <array>
         <dict>
           <key>match</key>
-          <string>(\\[nevr0zft"\\']|\\x[0-9a-fA-F][0-9a-fA-f])</string>
+          <string>(\\[nevr0zft"\\']|\\x[0-9a-fA-F]{2}|\\u[0-9a-fA-F]{4}|\\U[0-9a-fA-F]{6})</string>
           <key>name</key>
           <string>constant.character.escape.janet</string>
         </dict>