THIS PAGE IS PRELIMINARY AND REMAINS UNDER CONSTRUCTION. -Ian 9/13/2004


The Think Lisp Repository

1. Introduction

Welcome to the Think Lisp page. The various components of the Think project are based in one or more of C/C++, Python, Java and Lisp. However we feel that the flexibility provided by Lisp make it idea for prototyping and for the high-complexity components of our project. It is our desire that everyone in the group have a basic working competency with Lisp and at the very least we require that you know your way around enough to find and read code.

Lisp promotes a very different environment and development model than most other languages. Even Scheme, while sharing common syntax and a penchant for recursive expressions of interative constructs, does not have the same culture as Lisp. Lisp has often been equated with AI programming, functional programming and slow-execution and by many is considered a dead language. In reality lisp is alive and well, if less supported than many languages. There are even company success stories where smart people used Lisp to gain a major market advantage.

There are problems with Lisp, as there are with all languages, and Lisp's biggest challenge is keeping up with the ever changing protocols and sea of libraries being developed by the much larger set of developers in the open-source and in other languages world. The benefit of coming up to speed and maintaining proportionally more of your own code than the rest of the community is a massive increase in development speed and productivity. The serious student of lisp can accomplish more in less time than in any other language.

This site attempt to pull together a number of different resources, many of which are actively supported and used, to create a coherent environment with many of the capabilities and libraries one is accustomed to in modern language environments such as Python and Java. The site is served from a Lisp server ( CL-HTTPand much of the dynamic data on the web site comes directly from active Lisp objects inside the host image.

2. Learning Lisp

To learn lisp I recommend two great books by Paul Graham, ANSI Common Lisp and OnLisp. OnLisp in particular captures much of the essence of Lisp programming and is the best introduction to why one would want to program in this highly parenthetical language on a regular basis. OnLisp is out of print and can be downloaded here. There are also some relatively decent online books for learning lisp and symbolic computing from Franz Inc. and other sources. Familiarity with the material in MIT's 6.001 and the Sussman and Abelson text is always helpful. However I find Paul Graham to have the best pedagogy for already competent programmers. If you know nothing of Scheme or Lisp already, the Little Schemer is a good way to get a gentle, very basic, introduction.

I just discovered another online book I like that's about the be published that's more practically oriented towards Lisp as a software engineering language by Peter Siebel.

For accomplished programmers lisp can be rather annoying at first. There's the inevitable slowdown as you try accomodate your thinking and visual parsing to interpret the foreign syntax and unusual features. There are things going on behind the scenes in symbol interpretation and evaluation that are not present in the source code. Building a mental model of the reader, the macro-expansion phase and the evaluation phase and how a piece of textual syntax expresses itself in those stages of processing takes some doing.

There are more ways to accomplish things in Lisp, some better and some worse. It takes time to explore these variations and to learn how to craft clean lisp programs. Look at other's code, there is a culture around Lisp that is bigger and better than the language and while it's hard to find these days, it's worth looking at. Rather than writing my own arguments, I will point you to a set of essays by other folks trying to accomplish the same task.

2.1 What if I'm a Python programmer?

You're most of the way there. Python captures much of what makes Lisp a great language and in fact makes for a better scripting language than Lisp. Peter Norvig has a great overview of the differences between the two languages. Python is still less suited to large-scale systems development than Lisp, and the largest reason is the very reason Lisp has all those parentheses - macro systems. There are a number of other things that Lisp, as a 40 year old language, has picked up that Python is still working on but the other important characteristic is that Lisp is compiled and a well tuned Lisp program can compete for pure speed with the best C programs out there.

2.2 What if I'm a Java programmer?

Lisp has a very different style than Java, although you'll encounter familiar things like reference semantics, garbage collection and objects (called CLOS in Lisp, for the Common Lisp Object System). As with C programmers below, you're better off starting with a good tutorial that captures the Lisp culture such as ANSI Common Lisp. (NOTE: We'll add more resources here later)

2.3 What if I'm a C/C++ programmer?

You'll never have to call free or destroy or finalize ever again. More importantly when you get good at lisp you'll be typing 10 times less characters to achieve the same overall behavior as you did in C and ultimately saving several factors in total development time. You will pay modestly in performance, but many inner loops will run at the same speed with a little annotation (declare (type fixint foo)) but if you are desperate, you'll be able to import old and new C libraries into your lisp environment without with much less headache than writing C/C++ in the first place.

2.4 Major resources

There are a number of resources that may be of interest, rather than creating another index I will point you to a set of sites that you can peruse to find what you are looking for: There are also various introductions to using Lisp and thinking in Lisp which I will add to over time:

3 Tour of the Think System

The Think system consists of a source tree stored in a Subversion repository. You first need to download a local image of the repository and install a lisp that will work with the repository. We currently only support Franc, Inc's Allegro lisp environment but are moving towards a multi-platform environment that will work under MCL and SBCL. We also will try to keep things workable for Lispworks, CMUCL and Corman Lisp.

To edit code and talk to the lisp image interactively we use the Slime package which implements a protocol for having Emacs talk to a lisp image and provides a common front-end to many different lisp backends. Slime has both a lisp server and an emacs lisp package.

Many of the packages here are available on the net but have been added to the respository to ease the setup burden. If at any point a more recent version is available on the net inform the think codemaster and he or she will update the repository to reflect the latest version unless there's a good reason to do otherwise.

4 Setup and Getting Started

We support, roughly, Mac OS X, Windows/Cygwin and Linux platforms. We are moving to work predominantly on the Mac platform but maintain excellent support for Linux (this server runs on Debian) and lesser support for Windows/Cygwin because of Cygwin's flaky behavior in a number of areas.

4.1 Getting Around After Startup

Here we teach you a little about how to load things into your lisp image, what's loaded by default and how to perform some basic operations.

4.1.1 ASDF

To load individual subsystems we use the asdf (short for Another System Def Facility) system which has a simple, although annoyingly verbose syntax.
(asdf:operate 'asdf:load-op :systemname)
Will load all files associated with the defsystem file systemname.asd which should be located somewhere in lisp/, apps/, test/ or Roboverse/. The asd file is a lisp file defining the system, dependencies among files within the system and dependencies on other systems. A stable system will automatically load and import all the systems it depends on.

4.1.2 Think.Build:sync

We also have a little layer wrapped over asdf which allows you to setup a default environment. At any point your can type (think.build:sync) to update your system to the latest version of all your files. There are a number of systems defined in lisp/build such as think-core.asd, any of which can be assigned by name:
(setf think.build:*think-configuration* "think-core")
This is really just a synonym for (asdf:operate 'asdf:load-op :think-core).

4.2 Default Systems

think-core loads in the basic build systems that we assume exist in any image as a standard configuration. think-utils is another configuration that we presume exists in any running lisp image. It contains a set of standalone utility functions and systems such as regular expressions and lisp language extensions. Many of the Think system packages use these utilities so you want to be able to get around easily

4.3 Getting around - Packages

Critical to understanding how this system is organized is the discipline we employ in dealing with packages. NOTE: Add package policy here - in-package specifier - use-package specifier - Macros and quoted expression functions must be package independent

4.4 Using SVN

We have a particular way of using SVN that, while standard, requires slightly different command line hygene than you're probably used to. When messing around inside the svn heirarchy, always move, rename and delete files using svn so that the file history is maintained. When adding files you want saved in the repository use svn add to add them and make sure to commit then using svn commit. To find out what is committed and so on, use svn status.

svn add filename.lisp             # Adds the file to the repository on the next commit
svn commit filename.lisp          # Actually adds the file to the repository
svn remove filename.lisp          # Removes filename.lisp from the repository directory on the next comit
svn rename filename.lisp foo.lisp # Renames filename.lisp to foo.lisp and will track this change
                                  # on the next commit
svn move filename.lisp ../test/   # Moves the file to a new directory in the working copy, will do
                                    so in the repository on the last commit.
To permanently remove something and its history or if you are working on a package that many other people depend on talk to the codemaster about branches and more advanced svn hackery. Read the SVN manul for how to checkout an earlier revision and other cool things.

5 This Site as Reference

This site serves as a jumping off point to resources within the lisp system. Most of the facilities below are running in the image serving this website. The home page for each system will contain the following elements:

1) system name, defsystem type
2) packages declared and used
3) description of the system
4) List of the externally visible variables, functions and classes

Bookmark this reference page in your browser In lisp/trial we have a set of potentially useful systems that have not been tested or integrated into the mainstream yet, but we hold there for future importation.

NOTE: per package page template with ability to search docs, symbols, see code, etc.

6 An Introduction to Common Facilities

6.1 - Find your way around

- Web site has overview materials, can help you browse the system Utils has utilities for querying packages for names Utils has utilities for searching doc strings - Get Emacs up and running, use M-./M-, copiously - Use Paul Graham's books for basic utilities, search in utils for them (some names have changed for instance) - Pour through the following areas for basic examples of jumping off points for things you might want to do or extend.

6.2 - When things go wrong

Lisp has many facilities for handling exceptional control flow, either planned by the program, planned and informative exceptions, exceptions to the user, etc. The first thing you should know is that when an error occurs (car nil), you enter a debugger window prompt with a stack trace. C-h m gives you a list of all the keys supported by the current mode. 'n' and 'p' allow you to walk the stack frame. Often you can restart a frame by uwinding the stack to that point and reentering the computation. 'r' is a restart, 'R' returns a value from that stack frame. 't' will show local environment and 'l' (sometimes) will show you local variables.

Often there are options provided by default. Enter a value of the unbound variable, stop the computation, abort, continue processing, etc. Type the number of the option of interest to select it.

Debugging is easier when you run interpreted, and everything is available to the debugger. Use (asdf:operate 'asdf:load-source-op :system) instead of the usual load-op to evaluate the expressions in the source files of the package and dependant packages instead of compiling them.

There is an object inspector that allows you to follow references between objects and see all the elements in your program. In the REPL window or in a source window that refers to active programs, type 'C-c I' over the variable in question. This is particularily useful for CLOS classes, structures and list types with embedded structures.

6.4 - Testing Facilities

We have two testing models, one is a functional test suite for rapidly constructing a set of test cases for simple functional utilities and data structure manipulations in lisp/tests/function-testcase.lisp. The interface is simple; (use-package :think.test) in the package you are working in and then when you test a function, simply execute this form: (rftc (my-function arg1 arg2 arg3)) to register a functional test case. Later you can replay all your results by rerunning the arguments and comparing the new results to the original ones via (replay-test-cases 'my-function). If you use the keyword option :verbose t you will see the expressions printed as they are run. If you use :query-on-fail you can interactively strip out failing argument-result pairs that are no longer valid tests. You can save and load test cases by function or by specifying (save-test-cases :all). (replay-all-test-cases) will replay all saved test cases.
CL-USER> (defun foo (x y) (nconc (mapcar #'(lambda (x) (+ x 1)) x) (mapcar #'(lambda (x) (- x 1)) y)))
FOO
CL-USER> (rftc (foo '(1 2 3) '(1 2 3)))
(2 3 4 0 1 2)
CL-USER> (replay-test-cases 'foo)
T
CL-USER> (replay-test-cases 'foo :verbose t)
(FOO (1 2 3) (1 2 3)) -> (2 3 4 0 1 2)
T
CL-USER> (clear-test-cases 'foo)
NIL
CL-USER> (replay-test-cases 'foo :verbose t)
T
The replay-test-cases returns true even if there are no test cases. The aim of this package is to allow you to rapidly accumulate test cases as you interactively test functions. The save function saves the test cases as a lisp file so you can save it to your working directory and load it to validate code fixes later. The second method of testing uses more of a structured j-unit type of test where you have suites, cases and setup/teardown code. This is an important methdology for testing subsystems that have state. NOTE: Finish this section

6.3 - Common Operations and Interfaces

Examples of common, basic things you might want to do and pointers to applications or code that employ them.
- Basic algorithms and recursion
- Local databases and Non-class aggregate structures
- Time
- Access to the shell
- Files
- Database
- Network IO, URIs, XML-RPC
- Processes/MP
- External libraries and systems

6.4 Performance Monitoring and Tuning

6.5 Large packages

7 The Future

In many ways Lisp represents the future, and in many ways it is stuck in the past. Many of the modern facilities available in lisp are based on artifacts that came after the lisp heyday and are backwards in thought. At present we are stuck using this infrastructure but want to look forward towards new methodologies for managing complex, dynamic and ever-changing bodies of interacting code. Large scale code bases should have more of an ecology and less of a static design. The group is interested not just in extensions and models for systems engineering but the role that knowledge, language and social interactions play in the evolution and specification of large systems.

- Code repository and versioning
- Persistant code objects, global namespace, automatically extracted & managed dependencies (relaxing conservative assumptions) - Aggregate information across hosts; performance assumptions, module interactions, bug identification - Automated bug source discovery - Regression tracking from positive use examples (functional test suite) - Peer-to-peer infrastructure for robust global maintenance - Full history of code changes and source code changes in the system. - Rapid bootstrapping on new systems (export system description in global namespace)

- Semi-persistant Data Tracking?
- If you move to a new system, what data goes with you? How do we instrument a dynamic environment to make all that accessible? - State checkpoints. Low overhead ways to rollback to prior points in time and selectively import changes. History of major data transactions as well?

- High level OS interfaces
- OS and C wrapper methodologies - Database and transactional semantics; dynamic backend selection


Last modified: Thu Sep 23 17:27:40 Eastern Daylight Time 2004