cyclone/TODO

Roadmap:
 - Add constructs used by the compiler
 - Allow compiler to be hosted by Chicken (already working) or itself (self-hosting)
 - Once self-hosting, may want to create another repo to store snapshots of C code
 - Install scripts and instructions, need to be able to deploy this
   possibly install to /usr/local (allow specifying this on cmd line), similar to chibi
 - Add macro support (instead of current kludge)
 - Target r7rs support (coordinate with feature list)
 - User manual (or at least API docs)

Working TODO list:

 - self-hosting, there are a lot of accumulated TODO's that need to be addressed

   - improved error handling:
     - param count checks
       * should have checks in compiler as well. could possibly
         use prim metadata to autogenerate the runtime checks (later, perhaps).
         absolutely need this, but  may require some thought. which phase should
         do the checks? also may require looking at lib files because you need
         to check each application point, and need to look up the function to
         see how many args it can receive.
     - type checking
       ideally want to do this in a way that minimizes performance impacts.
       will probaby require extensive checks within apply() though, since that
       all happens at runtime.

     - error checking
       need errors for things such as file not existing, to bullet-proof the runtime
       some things to check for - file not found, divide by zero. what else?

     without these, it will be impossible (or at least time-consuming) to debug issues going forward

     2) Need to either allow code to read an import after macro expansion, or have another main module for self-hosting

   - GC improvements
     - is dhalloc limit good enough?
     - should expand size of heap if there is not enough room to fit all the "kept" objects

   - eval
     there is no concept of macro expansion, probably other deficiencies as well
     almost certainly will break when running the self-hosted compiler...

   - stack traces - not required but may be essential
     if we can record function names in cgen, then it is not too much
     of a stretch to create a circular buffer of string ptr's and then
     add to this buffer each time a function is called. then print the
     buffer out in the default exception handler (or such).

   - support for application of pure varargs functions, EG:
         ((lambda test (write test)) 1 2 3 4)
     don't think this is needed for self-hosting though, so it can wait

   - vectors
       - make-vector should have an optional 'fill' arg in compiled code
         note allocation functions can be functions instead of macros if they accept a
         cont arg so they do not have to return
       - should make GC more efficient, only transport mutated vector index, not the whole vector
       - After all this works, make sure to add tests from r7rs to test suite

   - I/O
     - may be able to use fmemopen to implement output strings, although it is not supported on windows

   - what else? there must be more stuff

 - Reduction in size of generated code
   is there anything we can do?
   are closures being packed/unpacked unnecessarily?

   right now, bundling this with below and attempting to optimize CPS conversion process to generate less code

 - Compiler efficiency

 Is there a more efficient way to CPS convert expressions? for example, can these primitive applications be inlined, instead of creating wrapping lambdas?

#;10> (cps-convert '(+ 1 (+ 2 2) (+ 3 3)))
 ((lambda (r$632) ((lambda (r$633) (%halt (+ 1 r$632 r$633))) (+ 3 3))) (+ 2 2))

 yes, but what can we do when a function is applied to more than just literals? are there still opportunities for optimization?
 may need to look through icyc.c for examples

 Are there instances where full CPS is not needed, and multiple expressions can be contained within a single lambda?

 - Simple idea about local variable elimination:

 http://www.pvk.ca/Blog/2012/02/19/fixed-points-and-strike-mandates/

 - Error handling
   need to perform much more error handling of input code. one of the biggest is to report if a function is passed the wrong number of arguments, as this will result in segfauls, bad transport errors, etc downstream if it is allowed.

  idea: set num_args and var_args in closure types. then can modify apply() to 
  check that the right number of args is provided for closure types. don't want to check
  numargs all the time, especially for compiled code. but this seems like a good compromise.

 - Add unit tests for exception handlers, and new constructs added to lib

 - Issues with detecting cycles:
   - (equal?) loops forever when comparing two different circular lists
   - printing *global-environment* in the repl still loops forever

 - Use a lib.scm for libs, similar to eval/parser modules?

 - along the same lines, output should be controlled in a better way. or at minimum should print errors using stderr (possible with standard scheme, though??)

 - String support
   issue is how to support strings themselves in memory. can add them directly to the string_type, but then apply won't work 
   because it could return an unknown number of bytes. on the other hand could use a separate data heap that is mirrored during GC.
   may need some extra buffer for that because technically it could overflow any time a new string is allocated, not just during
   function calls. but this would work for apply as well as everything else, I believe. obviously it makes GC a bit harder because
   there is another pair of heaps to deal with. but all that would be done is that strings from heap A would be copied to B during GC.
   GC would need to keep track of a pointer to each one. Sounds straightforward, but have to be careful of complications.
   Initial plan:
    - Add two "data" heap sections, and vars for each (head ptr, pos ptr [active only?], size)
    - Allocate string on active data heap via make_string
    - Initiate GC when stack exceeded or data heap under certain threshold
    - Need adequate extra space in data heap (100K? make config), since we only check it upon function call
    - Need to update GC to copy strings to other heap
    - Wait, this is broken if anything is pointing to one of these strings, since memory location changes upon GC!
      Is that a fatal issue? How to handle? could write string operations such that any operate on copies of
      strings rather than pointing to another string. not nearly as efficient but avoids this problem. could revisit
      other solutions down the road.
    - Anything else? Probably want to branch for this development, just in case there are complications

  COMPLICATION - only need to memcpy strings on data heap during a major collection. during a minor collection the strings are already where they need to be
  need to fully-implement this in the runtime by passing minor/major flag to transport

  TODO: trigger GC if data heap too low
  TODO: once this works but before moving all, consolidate all this in docs/strings.txt or such. would be useful to keep these notes


- Unit test improvements
  - concatenate all into one file when compiling / running
  - add assert functions, and actually test for equality
    otherwise it is too easy to miss failing test cases, unless they
    blow up the runtime
 - This has already been done, just need to incorporate other existing tests.

- Parser
  ;'a'c  ;; TODO: this is still an issue, try it

- Think about consoldating list of primitives in (prim?) and (c-compile-prim?). should also include other information such as number of args (or variable args), for error handling.

- Notes on implementing variables

 * lexical addressing (see chapter 5 of SICP) can be used to find a variable in recursive env's, so you can access it directly instead of having to recursively traverse environments.

- Implement ER-macros using eval, in scheme. with this in place, could implement basic macros using ER and replace "desugar". presumably syntax rules could be implemented this way as well, but that can wait for a later time
- Pass port along with emit procedures, to allow the scheme code to write to an output file (IE, c file)?? Or is with-output-file good enough? unfortunately that is not in husk yet so it makes boostrapping harder
- Question about closures and continuations:
 Presumably every function will recieve a closure. Do we have to differentiate between continuation (which every
 function must have) and closure (which can be empty if no fv)? right now the MTA runtime combines the two by
 having an fn argument to each closure. Is that OK?

- GC - notes from: http://www.more-magic.net/posts/internals-gc.html

 JAE - Good notes here about mutations (use write barrier to keep track of changes, EG: vector-set!). remember changes so they can be handled properly during next GC:

 Another major oversight is the assumption that objects can only point from the stack into the heap. If Scheme was a purely functional language, this would be entirely accurate: new objects can refer to old objects, but there is no way that a preexisting object can be made to refer to a newly created object. For that, you need to support mutation.
 But Scheme does support mutation! So what happens when you use vector-set! to store a newly created, stack-allocated value in an old, heap-allocated vector? If we used the above algorithm, the newly created element would either be part of the live set and get copied, but the vector's pointer would not be updated, or it wouldn't be part of the live set and the object would be lost in the stack reset.
 The answer to this problem is also pretty simple: we add a so-called write barrier. Whenever a value is written to an object, it is remembered. Then, when performing a GC, these remembered values are considered to be part of the live set, just like the addresses in the saved call. This is also the reason CHICKEN always shows the number of mutations when you're asking for GC statistics: mutation may slow down a program because GCs might take longer.

 JAE - Important point, that the heap must be reallocated during a major GC if there is too much data in the stack / old heap. Considered this but not sure if cyclone's GC does that right now:

The smart reader might have noticed a small problem here: what if the amount of garbage cleaned up is less than the data on the stack? Then, the stack data can't be copied to the new heap because it simply is too small. Well, this is when a third GC mode is triggered: a reallocating GC. This causes a new heap to be allocated, twice as big as the current heap. This is also split in from- and tospace. Then, Cheney's algorithm is performed on the old heap's fromspace, using one half of the new heap as tospace. When it's finished, the new tospace is called fromspace, and the other half of the new heap is called tospace. Then, the old heap is de-allocated.

- farther off but along the same lines, how to support compilation of
  multiple scheme files into multiple C modules?

- Just a thought: if this ever became self-hosting, could distribute compiled C files