janet/thoughts.md

Thoughts

A collection of thoughts and todo tasks for the project.

• Remove the concept of 'Ast node'. While providing fine-grained source mapping is is reasonably useful, it complicates the implementation of macros and other source transforming operations. Instead, we can key collection types (which have the unique pointer property) to source mapping information in a external data structure, which is specifically for source mapping. This data structure would be a hash table that used pointer equality instead of value equality for entries.

  The compiler would then need to keep track of 'most specific collection' node when compiling, as the value itself would not always be a collection and contain source mapping information.

  As a result, we could remove special forms like 'ast-quote', as well as ast-unwrapping logic which potentially duplicates a fair amount of data. Macros would be easier to write without needing to either unwrap ast values or sacrifice all source mapping.

• Serialization and deserialization of all datatypes. This would allow loading of bytecode without needing the compiler present. However, loading C functions is currently problematic. C functions could perhaps be wrapped in data structures that contain some meta information about them, say home module and types. This could also allow some automated type checking for C functions rather than writing it manually. Some slight overhead could perhaps be compensated for by adding optional ommission of typechecking later for C functions if it can be statically shown the types are sound.

• Better support for custom user datatypes. Tables and structs do work well for creating custom 'objects' and records, but lack ability to differentiate between object style values and data structure style values. For example, simply adding special keys as fields would make plain a table or struct possibly become object-like if the write keys are added.

  A Lua like solution would be metatables. It would perhaps make sense to only allow metatables on tables, as object like behavior seems to makes most sense on mutable data (?). For example, metatables on a struct could allow non-pure behavior unless they were extremely limited, or purity was enforced somehow. This could break expectations of a struct to behave as immutable data.

  Also, it might make sense that the metatable of a value would actually be a struct, so a metastruct. Implementations of Lua (LuaJIT) do not allow (or do not acknowledge) changing certain values of a metatables after it is set, such as gc, for performance reasons.

  Currently, tables can have a prototype table, which is used to look up values recursively if they are not found in the original table. This provides a partial solution to custom user types, but does not have the felxibility of Lua style metatables.

• Actually make a debugger. While the VM changes to enable debugging are relatively simple, make a useful debugger would be another project. At first, simply inspection of the generated bytecode assembly would be a good start. Single stepping, continuation, and inspecting the stack and current fiber would be a start.

  Preferably this would be programmed in dst. The needed functionality could be exposed in the fiber API.

  Later, with information from the compiler, we could inspect variables and upvalues by symbol. The current compiler does not do full SSA optimization, so named values are always accessible in the stack when in scope.

• Create a pool for fibers. Whlie the general purpose allocator and GC can be made more efficient, Fiber's can be well pooled because the allocated stack is large and can be reused. The stack size parameter on dst_fiber could simply become the minimum memory allocated for the stack. (Do a linear search throught the pool).

• Implement multi-methods.