cyclone/TODO

Roadmap:
 - Add macro support (instead of current kludge)
 - Target r7rs support (coordinate with feature list)
 - User manual (or at least API docs, features page may be a good 1st step)

Working TODO list. should start creating issues for these to get them out of here:

 - need to speed up the compilation cycle
   switch to using shared libraries instead of static ones
   move macro code to that module
   if necessary, write makefile directive to automate rebuilding cyclone for macro development

 - macros

   * thought: if we go with meta files, the compiler can revert back
     to using compiled macros if we do not install the meta files.
     just an idea, may be a hack or not feasible. not an awful idea to
     make them optional though, with some restrictions.

   how should eval deal with compiled macros?
   do we need a new tag type, eg: macro closure or such?
   then eval could say, if app macro, expand it and call analyze again
   may want to branch and experiment with this approach.
   or just add hooks, then experiment with it in eval...


   can start by adding define-syntax? section to transforms.scm.
   need to think about how macros from sld's will end up being
   loaded though. do we need to emit .meta files or such?

   how to load macros
  1- in same file, at compile time - works now with define-syntax
        there may be an issue here with top-level define's containing a macro that
        is contained in a top-level define-syntax listed after the define
  2- in same file, at runtime - would need to compile as a function, and mark somehow at runtime
        keep in mind macro expansion is only done via eval, so eval would need to have
        access to the compiled macros at runtime. with ER macros, it is very preferable
        for the macros to be compiled C functions
        * how do macros get into the global environment?

first, macros have to be converted to functions and compiled to even be available at runtime.
there will probably need to be a table of macros loaded at init time. the (macro?) function 
and others would use this table, much like they does now with *defined-macros*

each macro needs to tie a symbol (eg: 'let) to a compiled C function (or a closure??)

can we guarantee all libs are initiated before global environment?

  3- in library at compile time - how to load these??? do we need a ".meta" file or such???
      * basically the issue is we are compiling a file that references a macro in a library.
        the library is available as a .o and a .sld file at this point. but:
          a) how do we know an exported object is a macro, and
          b) how do we get the macro function (either compiled or as a scheme AST)?

        one possible solution is to emit an intermediate file containing a list of
        macros, like *defined-macros*.

on the other hand, how is this going to work when compiling cyclone? obviously want the
compiler to use compiled functions as much as possible. if there is a naming conflict
can it just use the compiled one by default? don't want cyclone eval'ing functions
when it does not have to, and slowing down compilation even more

      Previous notes:
      * library is compiled, but how to know what the macros are, and how to access them?
        could emit a meta file that contains the macro identifiers and either their body
        or a way to access them in the obj file (may not be possible/portable)
  4- in library at runtime - if marked as macro, could run function (should be nice and fast)
     assume this would be done in the same manner (or similar) to (2).
   - anything else?



 * macros within macros, 4 cases:
   TODO: explain how each one will be expanded:
   - compiled macro within compiled macro
   - eval'd macro within compiled macro
   - compiled macro within eval'd macro
   - eval'd macro within eval'd macro





 - profile cyclone code, see if there are any bottlenecks we can reduce
   EG: prof cyclone transforms.sld

 - self-hosting, there are a lot of accumulated TODO's that need to be addressed

   - adding r7rs support
     - review other sections from the report

   - improved error handling:
     - param count checks
       if a primitive is called directly, shouldn't it be possible to check arg count?
       is what we have now robust enough to prevent segfaults?

     - type checking
       done for now, check performance compiling transforms.sld

     2) Need to either allow code to read an import after macro expansion, or have another main module for self-hosting

   - what's going on here:
   cyclone> (call/cc (lambda (k) (k 1)))
   Error: Expected 2 arguments but received 1.

   - eval
     there is limited concept of macro expansion, probably other deficiencies as well
     call/cc issues?
     almost certainly will break when running the self-hosted compiler...
     no, I think this is ok. may want to unify macros with compiler side though

   - I/O
     - may be able to use fmemopen to implement output strings, although it is not supported on windows

   - what else? there must be more stuff

 - Reduction in size of generated code
   is there anything we can do?
   are closures being packed/unpacked unnecessarily?

   right now, bundling this with below and attempting to optimize CPS conversion process to generate less code

 - Compiler efficiency

 Is there a more efficient way to CPS convert expressions? for example, can these primitive applications be inlined, instead of creating wrapping lambdas?

#;10> (cps-convert '(+ 1 (+ 2 2) (+ 3 3)))
 ((lambda (r$632) ((lambda (r$633) (%halt (+ 1 r$632 r$633))) (+ 3 3))) (+ 2 2))

 yes, but what can we do when a function is applied to more than just literals? are there still opportunities for optimization?
 may need to look through icyc.c for examples

 Are there instances where full CPS is not needed, and multiple expressions can be contained within a single lambda?

 - Simple idea about local variable elimination:

 http://www.pvk.ca/Blog/2012/02/19/fixed-points-and-strike-mandates/

 - String support
   issue is how to support strings themselves in memory. can add them directly to the string_type, but then apply won't work 
   because it could return an unknown number of bytes. on the other hand could use a separate data heap that is mirrored during GC.
   may need some extra buffer for that because technically it could overflow any time a new string is allocated, not just during
   function calls. but this would work for apply as well as everything else, I believe. obviously it makes GC a bit harder because
   there is another pair of heaps to deal with. but all that would be done is that strings from heap A would be copied to B during GC.
   GC would need to keep track of a pointer to each one. Sounds straightforward, but have to be careful of complications.
   Initial plan:
    - Add two "data" heap sections, and vars for each (head ptr, pos ptr [active only?], size)
    - Allocate string on active data heap via make_string
    - Initiate GC when stack exceeded or data heap under certain threshold
    - Need adequate extra space in data heap (100K? make config), since we only check it upon function call
    - Need to update GC to copy strings to other heap
    - Wait, this is broken if anything is pointing to one of these strings, since memory location changes upon GC!
      Is that a fatal issue? How to handle? could write string operations such that any operate on copies of
      strings rather than pointing to another string. not nearly as efficient but avoids this problem. could revisit
      other solutions down the road.
    - Anything else? Probably want to branch for this development, just in case there are complications

  COMPLICATION - only need to memcpy strings on data heap during a major collection. during a minor collection the strings are already where they need to be
  need to fully-implement this in the runtime by passing minor/major flag to transport

  TODO: trigger GC if data heap too low
  TODO: once this works but before moving all, consolidate all this in docs/strings.txt or such. would be useful to keep these notes

- Think about consoldating list of primitives in (prim?) and (c-compile-prim?). should also include other information such as number of args (or variable args), for error handling.

- Notes on implementing variables

 * lexical addressing (see chapter 5 of SICP) can be used to find a variable in recursive env's, so you can access it directly instead of having to recursively traverse environments.

- Question about closures and continuations:
 Presumably every function will recieve a closure. Do we have to differentiate between continuation (which every
 function must have) and closure (which can be empty if no fv)? right now the MTA runtime combines the two by
 having an fn argument to each closure. Is that OK?

- keeping this in here, should mention this in the 'how this works' doc:
GC - notes from: http://www.more-magic.net/posts/internals-gc.html

 JAE - Good notes here about mutations (use write barrier to keep track of changes, EG: vector-set!). remember changes so they can be handled properly during next GC:

 Another major oversight is the assumption that objects can only point from the stack into the heap. If Scheme was a purely functional language, this would be entirely accurate: new objects can refer to old objects, but there is no way that a preexisting object can be made to refer to a newly created object. For that, you need to support mutation.
 But Scheme does support mutation! So what happens when you use vector-set! to store a newly created, stack-allocated value in an old, heap-allocated vector? If we used the above algorithm, the newly created element would either be part of the live set and get copied, but the vector's pointer would not be updated, or it wouldn't be part of the live set and the object would be lost in the stack reset.
 The answer to this problem is also pretty simple: we add a so-called write barrier. Whenever a value is written to an object, it is remembered. Then, when performing a GC, these remembered values are considered to be part of the live set, just like the addresses in the saved call. This is also the reason CHICKEN always shows the number of mutations when you're asking for GC statistics: mutation may slow down a program because GCs might take longer.

 JAE - Important point, that the heap must be reallocated during a major GC if there is too much data in the stack / old heap. Considered this but not sure if cyclone's GC does that right now:

The smart reader might have noticed a small problem here: what if the amount of garbage cleaned up is less than the data on the stack? Then, the stack data can't be copied to the new heap because it simply is too small. Well, this is when a third GC mode is triggered: a reallocating GC. This causes a new heap to be allocated, twice as big as the current heap. This is also split in from- and tospace. Then, Cheney's algorithm is performed on the old heap's fromspace, using one half of the new heap as tospace. When it's finished, the new tospace is called fromspace, and the other half of the new heap is called tospace. Then, the old heap is de-allocated.