📜 Add better documentation across the compiler. #3
@@ -1,28 +1,28 @@
|
|||||||
//! # The compiler backend: generation of machine code, both static and JIT.
|
//! # The compiler backend: generation of machine code, both static and JIT.
|
||||||
//!
|
//!
|
||||||
//! This module is responsible for taking our intermediate representation from
|
//! This module is responsible for taking our intermediate representation from
|
||||||
//! [`crate::ir`] and turning it into Cranelift and then into object code that
|
//! [`crate::ir`] and turning it into Cranelift and then into object code that
|
||||||
//! can either be saved to disk or run in memory. Because the runtime functions
|
//! can either be saved to disk or run in memory. Because the runtime functions
|
||||||
//! for NGR are very closely tied to the compiler implentation, we also include
|
//! for NGR are very closely tied to the compiler implentation, we also include
|
||||||
//! information about these functions as part of the module.
|
//! information about these functions as part of the module.
|
||||||
//!
|
//!
|
||||||
//! ## Using the `Backend`
|
//! ## Using the `Backend`
|
||||||
//!
|
//!
|
||||||
//! The backend of this compiler can be used in two modes: a static compilation
|
//! The backend of this compiler can be used in two modes: a static compilation
|
||||||
//! mode, where the goal is to write the compiled object to disk and then link
|
//! mode, where the goal is to write the compiled object to disk and then link
|
||||||
//! it later, and a JIT mode, where the goal is to write the compiled object to
|
//! it later, and a JIT mode, where the goal is to write the compiled object to
|
||||||
//! memory and then run it. Both modes use the same `Backend` object, because
|
//! memory and then run it. Both modes use the same `Backend` object, because
|
||||||
//! they share a lot of behaviors. However, you'll want to use different variants
|
//! they share a lot of behaviors. However, you'll want to use different variants
|
||||||
//! based on your goals:
|
//! based on your goals:
|
||||||
//!
|
//!
|
||||||
//! * Use `Backend<ObjectModule>`, constructed via [`Backend::object_file`],
|
//! * Use `Backend<ObjectModule>`, constructed via [`Backend::object_file`],
|
||||||
//! if you want to compile to an object file on disk, which you're then going
|
//! if you want to compile to an object file on disk, which you're then going
|
||||||
//! to link to later.
|
//! to link to later.
|
||||||
//! * Use `Backend<JITModule>`, constructed via [`Backend::jit`], if you want
|
//! * Use `Backend<JITModule>`, constructed via [`Backend::jit`], if you want
|
||||||
//! to do just-in-time compilation and are just going to run things immediately.
|
//! to do just-in-time compilation and are just going to run things immediately.
|
||||||
//!
|
//!
|
||||||
//! ## Working with Runtime Functions
|
//! ## Working with Runtime Functions
|
||||||
//!
|
//!
|
||||||
//! For now, runtime functions are pretty easy to describe, because there's
|
//! For now, runtime functions are pretty easy to describe, because there's
|
||||||
//! only one. In the future, though, the [`RuntimeFunctions`] object is there to
|
//! only one. In the future, though, the [`RuntimeFunctions`] object is there to
|
||||||
//! help provide a clean interface to them all.
|
//! help provide a clean interface to them all.
|
||||||
@@ -45,7 +45,7 @@ use target_lexicon::Triple;
|
|||||||
const EMPTY_DATUM: [u8; 8] = [0; 8];
|
const EMPTY_DATUM: [u8; 8] = [0; 8];
|
||||||
|
|
||||||
/// An object representing an active backend.
|
/// An object representing an active backend.
|
||||||
///
|
///
|
||||||
/// Internally, this object holds a bunch of state useful for compiling one
|
/// Internally, this object holds a bunch of state useful for compiling one
|
||||||
/// or more functions into an object file or memory. It can be passed around,
|
/// or more functions into an object file or memory. It can be passed around,
|
||||||
/// but cannot currently be duplicated because some of that state is not
|
/// but cannot currently be duplicated because some of that state is not
|
||||||
@@ -64,7 +64,7 @@ pub struct Backend<M: Module> {
|
|||||||
|
|
||||||
impl Backend<JITModule> {
|
impl Backend<JITModule> {
|
||||||
/// Create a new JIT backend for compiling NGR into memory.
|
/// Create a new JIT backend for compiling NGR into memory.
|
||||||
///
|
///
|
||||||
/// The provided output buffer is not for the compiled code, but for the output
|
/// The provided output buffer is not for the compiled code, but for the output
|
||||||
/// of any `print` expressions that are evaluated. If set to `None`, the output
|
/// of any `print` expressions that are evaluated. If set to `None`, the output
|
||||||
/// will be written to `stdout` as per normal, but if a String buffer is provided,
|
/// will be written to `stdout` as per normal, but if a String buffer is provided,
|
||||||
@@ -95,7 +95,7 @@ impl Backend<JITModule> {
|
|||||||
|
|
||||||
/// Given a compiled function ID, get a pointer to where that function was written
|
/// Given a compiled function ID, get a pointer to where that function was written
|
||||||
/// in memory.
|
/// in memory.
|
||||||
///
|
///
|
||||||
/// The data at this pointer should not be mutated unless you really, really,
|
/// The data at this pointer should not be mutated unless you really, really,
|
||||||
/// really know what you're doing. It can be run by casting it into a Rust
|
/// really know what you're doing. It can be run by casting it into a Rust
|
||||||
/// `fn() -> ()`, and then calling it from normal Rust.
|
/// `fn() -> ()`, and then calling it from normal Rust.
|
||||||
@@ -106,7 +106,7 @@ impl Backend<JITModule> {
|
|||||||
|
|
||||||
impl Backend<ObjectModule> {
|
impl Backend<ObjectModule> {
|
||||||
/// Generate a backend for compiling into an object file for the given target.
|
/// Generate a backend for compiling into an object file for the given target.
|
||||||
///
|
///
|
||||||
/// This backend will generate a single output file per `Backend` object, although
|
/// This backend will generate a single output file per `Backend` object, although
|
||||||
/// that file may have multiple functions defined within it. Data between those
|
/// that file may have multiple functions defined within it. Data between those
|
||||||
/// functions (in particular, strings) will be defined once and shared between
|
/// functions (in particular, strings) will be defined once and shared between
|
||||||
@@ -139,11 +139,11 @@ impl Backend<ObjectModule> {
|
|||||||
|
|
||||||
impl<M: Module> Backend<M> {
|
impl<M: Module> Backend<M> {
|
||||||
/// Define a string within the current backend.
|
/// Define a string within the current backend.
|
||||||
///
|
///
|
||||||
/// Note that this is a Cranelift [`DataId`], which then must be redeclared inside the
|
/// Note that this is a Cranelift [`DataId`], which then must be redeclared inside the
|
||||||
/// context of any functions or data items that want to use it. That being said, the
|
/// context of any functions or data items that want to use it. That being said, the
|
||||||
/// string value will be defined once in the file and then shared by all referencers.
|
/// string value will be defined once in the file and then shared by all referencers.
|
||||||
///
|
///
|
||||||
/// This function will automatically add a null character (`'\0'`) to the end of the
|
/// This function will automatically add a null character (`'\0'`) to the end of the
|
||||||
/// string, to ensure that strings are non-terminated for interactions with other
|
/// string, to ensure that strings are non-terminated for interactions with other
|
||||||
/// languages.
|
/// languages.
|
||||||
@@ -163,7 +163,7 @@ impl<M: Module> Backend<M> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Define a global variable within the current backend.
|
/// Define a global variable within the current backend.
|
||||||
///
|
///
|
||||||
/// These variables can be shared between functions, and will be exported from the
|
/// These variables can be shared between functions, and will be exported from the
|
||||||
/// module itself as public data in the case of static compilation. There initial
|
/// module itself as public data in the case of static compilation. There initial
|
||||||
/// value will be null.
|
/// value will be null.
|
||||||
@@ -179,7 +179,7 @@ impl<M: Module> Backend<M> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Get a pointer to the output buffer for `print`ing, or `null`.
|
/// Get a pointer to the output buffer for `print`ing, or `null`.
|
||||||
///
|
///
|
||||||
/// As suggested, returns `null` in the case where the user has not provided an
|
/// As suggested, returns `null` in the case where the user has not provided an
|
||||||
/// output buffer; it is your responsibility to check for this case and do
|
/// output buffer; it is your responsibility to check for this case and do
|
||||||
/// something sensible.
|
/// something sensible.
|
||||||
@@ -192,7 +192,7 @@ impl<M: Module> Backend<M> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Get any captured output `print`ed by the program during execution.
|
/// Get any captured output `print`ed by the program during execution.
|
||||||
///
|
///
|
||||||
/// If an output buffer was not provided, or if the program has not done any
|
/// If an output buffer was not provided, or if the program has not done any
|
||||||
/// printing, then this function will return an empty string.
|
/// printing, then this function will return an empty string.
|
||||||
pub fn output(self) -> String {
|
pub fn output(self) -> String {
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ use pretty::termcolor::{ColorChoice, StandardStream};
|
|||||||
use target_lexicon::Triple;
|
use target_lexicon::Triple;
|
||||||
|
|
||||||
/// A high-level compiler for NGR programs.
|
/// A high-level compiler for NGR programs.
|
||||||
///
|
///
|
||||||
/// This object can be built once, and then re-used many times to build multiple
|
/// This object can be built once, and then re-used many times to build multiple
|
||||||
/// files. For most users, the [`Default`] implementation should be sufficient;
|
/// files. For most users, the [`Default`] implementation should be sufficient;
|
||||||
/// it will use `stderr` for warnings and errors, with default colors based on
|
/// it will use `stderr` for warnings and errors, with default colors based on
|
||||||
@@ -62,17 +62,17 @@ impl Compiler {
|
|||||||
|
|
||||||
/// This is the actual meat of the compilation chain; we hide it from the user
|
/// This is the actual meat of the compilation chain; we hide it from the user
|
||||||
/// because the type is kind of unpleasant.
|
/// because the type is kind of unpleasant.
|
||||||
///
|
///
|
||||||
/// The weird error type comes from the fact that we can run into three types
|
/// The weird error type comes from the fact that we can run into three types
|
||||||
/// of result:
|
/// of result:
|
||||||
///
|
///
|
||||||
/// * Fundamental errors, like an incorrectly formatted file or some
|
/// * Fundamental errors, like an incorrectly formatted file or some
|
||||||
/// oddity with IO. These return `Err`.
|
/// oddity with IO. These return `Err`.
|
||||||
/// * Validation errors, where we reject the program due to something
|
/// * Validation errors, where we reject the program due to something
|
||||||
/// semantically wrong with them. These return `Ok(None)`.
|
/// semantically wrong with them. These return `Ok(None)`.
|
||||||
/// * Success! In this case, we return `Ok(Some(...))`, where the bytes
|
/// * Success! In this case, we return `Ok(Some(...))`, where the bytes
|
||||||
/// returned is the contents of the compiled object file.
|
/// returned is the contents of the compiled object file.
|
||||||
///
|
///
|
||||||
fn compile_internal(&mut self, input_file: &str) -> Result<Option<Vec<u8>>, CompilerError> {
|
fn compile_internal(&mut self, input_file: &str) -> Result<Option<Vec<u8>>, CompilerError> {
|
||||||
// Try to parse the file into our syntax AST. If we fail, emit the error
|
// Try to parse the file into our syntax AST. If we fail, emit the error
|
||||||
// and then immediately return `None`.
|
// and then immediately return `None`.
|
||||||
@@ -111,7 +111,7 @@ impl Compiler {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Emit a diagnostic.
|
/// Emit a diagnostic.
|
||||||
///
|
///
|
||||||
/// This is just a really handy shorthand we use elsewhere in the object, because
|
/// This is just a really handy shorthand we use elsewhere in the object, because
|
||||||
/// there's a lot of boilerplate we'd like to skip.
|
/// there's a lot of boilerplate we'd like to skip.
|
||||||
fn emit(&mut self, diagnostic: Diagnostic<usize>) {
|
fn emit(&mut self, diagnostic: Diagnostic<usize>) {
|
||||||
|
|||||||
16
src/eval.rs
16
src/eval.rs
@@ -1,5 +1,5 @@
|
|||||||
//! Helpful functions for evaluating NGR programs.
|
//! Helpful functions for evaluating NGR programs.
|
||||||
//!
|
//!
|
||||||
//! Look, this is a compiler, and so you might be asking why it has a bunch of
|
//! Look, this is a compiler, and so you might be asking why it has a bunch of
|
||||||
//! stuff in it to help with writing interpreters. Well, the answer is simple:
|
//! stuff in it to help with writing interpreters. Well, the answer is simple:
|
||||||
//! testing. It's really nice to know that if you start with a program that
|
//! testing. It's really nice to know that if you start with a program that
|
||||||
@@ -9,15 +9,15 @@
|
|||||||
//! programs don't do 100% the same things in the same order, but you shouldn't
|
//! programs don't do 100% the same things in the same order, but you shouldn't
|
||||||
//! be able to observe the difference ... at least, not without a stopwatch,
|
//! be able to observe the difference ... at least, not without a stopwatch,
|
||||||
//! memory profilers, etc.
|
//! memory profilers, etc.
|
||||||
//!
|
//!
|
||||||
//! The actual evaluators for our various syntaxes are hidden in `eval` functions
|
//! The actual evaluators for our various syntaxes are hidden in `eval` functions
|
||||||
//! of the various ASTs. It's nice to have them "next to" the syntax that way, so
|
//! of the various ASTs. It's nice to have them "next to" the syntax that way, so
|
||||||
//! that we just edit stuff in one part of the source tree at a time. This module,
|
//! that we just edit stuff in one part of the source tree at a time. This module,
|
||||||
//! then, just contains some things that are generally helpful across all the
|
//! then, just contains some things that are generally helpful across all the
|
||||||
//! interpreters we've written.
|
//! interpreters we've written.
|
||||||
//!
|
//!
|
||||||
//! In particular, this module helps with:
|
//! In particular, this module helps with:
|
||||||
//!
|
//!
|
||||||
//! * Defining a common error type -- [`EvalError`] -- that we can reasonably
|
//! * Defining a common error type -- [`EvalError`] -- that we can reasonably
|
||||||
//! compare. It's nice to compare errors, here, because we want to know that
|
//! compare. It's nice to compare errors, here, because we want to know that
|
||||||
//! if a program used to fail, it will still fail after we change it, and
|
//! if a program used to fail, it will still fail after we change it, and
|
||||||
@@ -45,7 +45,7 @@ pub use value::Value;
|
|||||||
use crate::backend::BackendError;
|
use crate::backend::BackendError;
|
||||||
|
|
||||||
/// All of the errors that can happen trying to evaluate an NGR program.
|
/// All of the errors that can happen trying to evaluate an NGR program.
|
||||||
///
|
///
|
||||||
/// This is yet another standard [`thiserror::Error`] type, but with the
|
/// This is yet another standard [`thiserror::Error`] type, but with the
|
||||||
/// caveat that it implements [`PartialEq`] even though some of its
|
/// caveat that it implements [`PartialEq`] even though some of its
|
||||||
/// constituent members don't. It does so through the very sketchy mechanism
|
/// constituent members don't. It does so through the very sketchy mechanism
|
||||||
@@ -102,17 +102,17 @@ impl PartialEq for EvalError {
|
|||||||
EvalError::Linker(a) => match other {
|
EvalError::Linker(a) => match other {
|
||||||
EvalError::Linker(b) => a == b,
|
EvalError::Linker(b) => a == b,
|
||||||
_ => false,
|
_ => false,
|
||||||
}
|
},
|
||||||
|
|
||||||
EvalError::ExitCode(a) => match other {
|
EvalError::ExitCode(a) => match other {
|
||||||
EvalError::ExitCode(b) => a == b,
|
EvalError::ExitCode(b) => a == b,
|
||||||
_ => false,
|
_ => false,
|
||||||
}
|
},
|
||||||
|
|
||||||
EvalError::RuntimeOutput(a) => match other {
|
EvalError::RuntimeOutput(a) => match other {
|
||||||
EvalError::RuntimeOutput(b) => a == b,
|
EvalError::RuntimeOutput(b) => a == b,
|
||||||
_ => false,
|
_ => false,
|
||||||
}
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ use std::sync::Arc;
|
|||||||
|
|
||||||
/// An evaluation environment, which maps variable names to their
|
/// An evaluation environment, which maps variable names to their
|
||||||
/// current values.
|
/// current values.
|
||||||
///
|
///
|
||||||
/// One key difference between `EvalEnvironment` and `HashMap` is that
|
/// One key difference between `EvalEnvironment` and `HashMap` is that
|
||||||
/// `EvalEnvironment` uses an `extend` mechanism to add keys, rather
|
/// `EvalEnvironment` uses an `extend` mechanism to add keys, rather
|
||||||
/// than an `insert`. This difference allows you to add mappings for
|
/// than an `insert`. This difference allows you to add mappings for
|
||||||
@@ -20,7 +20,7 @@ enum EvalEnvInternal {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Errors that can happen when looking up a variable.
|
/// Errors that can happen when looking up a variable.
|
||||||
///
|
///
|
||||||
/// This enumeration may be extended in the future, depending on if we
|
/// This enumeration may be extended in the future, depending on if we
|
||||||
/// get more subtle with our keys. But for now, this is just a handy
|
/// get more subtle with our keys. But for now, this is just a handy
|
||||||
/// way to make lookup failures be `thiserror::Error`s.
|
/// way to make lookup failures be `thiserror::Error`s.
|
||||||
@@ -45,7 +45,7 @@ impl EvalEnvironment {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Extend the environment with a new mapping.
|
/// Extend the environment with a new mapping.
|
||||||
///
|
///
|
||||||
/// Note the types: the result of this method is a new `EvalEnvironment`,
|
/// Note the types: the result of this method is a new `EvalEnvironment`,
|
||||||
/// with its own lifetime, and the original environment is left unmodified.
|
/// with its own lifetime, and the original environment is left unmodified.
|
||||||
pub fn extend(&self, name: ArcIntern<String>, value: Value) -> Self {
|
pub fn extend(&self, name: ArcIntern<String>, value: Value) -> Self {
|
||||||
|
|||||||
@@ -75,7 +75,7 @@ impl Value {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Calculate the result of running the given primitive on the given arguments.
|
/// Calculate the result of running the given primitive on the given arguments.
|
||||||
///
|
///
|
||||||
/// This can cause errors in a whole mess of ways, so be careful about your
|
/// This can cause errors in a whole mess of ways, so be careful about your
|
||||||
/// inputs. For example, addition only works when the two values have the exact
|
/// inputs. For example, addition only works when the two values have the exact
|
||||||
/// same type, so expect an error if you try to do so. In addition, this
|
/// same type, so expect an error if you try to do so. In addition, this
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
use std::fmt::Display;
|
use std::fmt::Display;
|
||||||
|
|
||||||
/// Values in the interpreter.
|
/// Values in the interpreter.
|
||||||
///
|
///
|
||||||
/// Yes, this is yet another definition of a structure called `Value`, which
|
/// Yes, this is yet another definition of a structure called `Value`, which
|
||||||
/// are almost entirely identical. However, it's nice to have them separated
|
/// are almost entirely identical. However, it's nice to have them separated
|
||||||
/// by type so that we don't mix them up.
|
/// by type so that we don't mix them up.
|
||||||
|
|||||||
@@ -1,12 +1,12 @@
|
|||||||
//! The middle of the compiler: analysis, simplification, optimization.
|
//! The middle of the compiler: analysis, simplification, optimization.
|
||||||
//!
|
//!
|
||||||
//! For the moment, this module doesn't do much besides define an intermediate
|
//! For the moment, this module doesn't do much besides define an intermediate
|
||||||
//! representation for NGR programs that is a little easier to work with then
|
//! representation for NGR programs that is a little easier to work with then
|
||||||
//! the structures we've built from the actual user syntax. For example, in the
|
//! the structures we've built from the actual user syntax. For example, in the
|
||||||
//! IR syntax, function calls are simplified so that all their arguments are
|
//! IR syntax, function calls are simplified so that all their arguments are
|
||||||
//! either variables or constants, which can make reasoning about programs
|
//! either variables or constants, which can make reasoning about programs
|
||||||
//! (and implicit temporary variables) quite a bit easier.
|
//! (and implicit temporary variables) quite a bit easier.
|
||||||
//!
|
//!
|
||||||
//! For the foreseeable future, this module will likely remain mostly empty
|
//! For the foreseeable future, this module will likely remain mostly empty
|
||||||
//! besides definitions, as we'll likely want to focus on just processing /
|
//! besides definitions, as we'll likely want to focus on just processing /
|
||||||
//! validating syntax, and then figuring out how to turn it into Cranelift
|
//! validating syntax, and then figuring out how to turn it into Cranelift
|
||||||
|
|||||||
@@ -7,17 +7,17 @@ use proptest::{
|
|||||||
};
|
};
|
||||||
|
|
||||||
/// We're going to represent variables as interned strings.
|
/// We're going to represent variables as interned strings.
|
||||||
///
|
///
|
||||||
/// These should be fast enough for comparison that it's OK, since it's going to end up
|
/// These should be fast enough for comparison that it's OK, since it's going to end up
|
||||||
/// being pretty much the pointer to the string.
|
/// being pretty much the pointer to the string.
|
||||||
type Variable = ArcIntern<String>;
|
type Variable = ArcIntern<String>;
|
||||||
|
|
||||||
/// The representation of a program within our IR. For now, this is exactly one file.
|
/// The representation of a program within our IR. For now, this is exactly one file.
|
||||||
///
|
///
|
||||||
/// In addition, for the moment there's not really much of interest to hold here besides
|
/// In addition, for the moment there's not really much of interest to hold here besides
|
||||||
/// the list of statements read from the file. Order is important. In the future, you
|
/// the list of statements read from the file. Order is important. In the future, you
|
||||||
/// could imagine caching analysis information in this structure.
|
/// could imagine caching analysis information in this structure.
|
||||||
///
|
///
|
||||||
/// `Program` implements both [`Pretty`] and [`Arbitrary`]. The former should be used
|
/// `Program` implements both [`Pretty`] and [`Arbitrary`]. The former should be used
|
||||||
/// to print the structure whenever possible, especially if you value your or your
|
/// to print the structure whenever possible, especially if you value your or your
|
||||||
/// user's time. The latter is useful for testing that conversions of `Program` retain
|
/// user's time. The latter is useful for testing that conversions of `Program` retain
|
||||||
@@ -63,15 +63,15 @@ impl Arbitrary for Program {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// The representation of a statement in the language.
|
/// The representation of a statement in the language.
|
||||||
///
|
///
|
||||||
/// For now, this is either a binding site (`x = 4`) or a print statement
|
/// For now, this is either a binding site (`x = 4`) or a print statement
|
||||||
/// (`print x`). Someday, though, more!
|
/// (`print x`). Someday, though, more!
|
||||||
///
|
///
|
||||||
/// As with `Program`, this type implements [`Pretty`], which should
|
/// As with `Program`, this type implements [`Pretty`], which should
|
||||||
/// be used to display the structure whenever possible. It does not
|
/// be used to display the structure whenever possible. It does not
|
||||||
/// implement [`Arbitrary`], though, mostly because it's slightly
|
/// implement [`Arbitrary`], though, mostly because it's slightly
|
||||||
/// complicated to do so.
|
/// complicated to do so.
|
||||||
///
|
///
|
||||||
#[derive(Debug)]
|
#[derive(Debug)]
|
||||||
pub enum Statement {
|
pub enum Statement {
|
||||||
Binding(Location, Variable, Expression),
|
Binding(Location, Variable, Expression),
|
||||||
@@ -100,11 +100,11 @@ where
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// The representation of an expression.
|
/// The representation of an expression.
|
||||||
///
|
///
|
||||||
/// Note that expressions, like everything else in this syntax tree,
|
/// Note that expressions, like everything else in this syntax tree,
|
||||||
/// supports [`Pretty`], and it's strongly encouraged that you use
|
/// supports [`Pretty`], and it's strongly encouraged that you use
|
||||||
/// that trait/module when printing these structures.
|
/// that trait/module when printing these structures.
|
||||||
///
|
///
|
||||||
/// Also, Expressions at this point in the compiler are explicitly
|
/// Also, Expressions at this point in the compiler are explicitly
|
||||||
/// defined so that they are *not* recursive. By this point, if an
|
/// defined so that they are *not* recursive. By this point, if an
|
||||||
/// expression requires some other data (like, for example, invoking
|
/// expression requires some other data (like, for example, invoking
|
||||||
@@ -148,7 +148,7 @@ where
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// A type representing the primitives allowed in the language.
|
/// A type representing the primitives allowed in the language.
|
||||||
///
|
///
|
||||||
/// Having this as an enumeration avoids a lot of "this should not happen"
|
/// Having this as an enumeration avoids a lot of "this should not happen"
|
||||||
/// cases, but might prove to be cumbersome in the future. If that happens,
|
/// cases, but might prove to be cumbersome in the future. If that happens,
|
||||||
/// this may either become a more hierarchical enumeration, or we'll just
|
/// this may either become a more hierarchical enumeration, or we'll just
|
||||||
@@ -191,7 +191,7 @@ where
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// An expression that is always either a value or a reference.
|
/// An expression that is always either a value or a reference.
|
||||||
///
|
///
|
||||||
/// This is the type used to guarantee that we don't nest expressions
|
/// This is the type used to guarantee that we don't nest expressions
|
||||||
/// at this level. Instead, expressions that take arguments take one
|
/// at this level. Instead, expressions that take arguments take one
|
||||||
/// of these, which can only be a constant or a reference.
|
/// of these, which can only be a constant or a reference.
|
||||||
@@ -227,7 +227,7 @@ impl From<ValueOrRef> for Expression {
|
|||||||
#[derive(Debug)]
|
#[derive(Debug)]
|
||||||
pub enum Value {
|
pub enum Value {
|
||||||
/// A numerical constant.
|
/// A numerical constant.
|
||||||
///
|
///
|
||||||
/// The optional argument is the base that was used by the user to input
|
/// The optional argument is the base that was used by the user to input
|
||||||
/// the number. By retaining it, we can ensure that if we need to print the
|
/// the number. By retaining it, we can ensure that if we need to print the
|
||||||
/// number back out, we can do so in the form that the user entered it.
|
/// number back out, we can do so in the form that the user entered it.
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ use super::{Primitive, ValueOrRef};
|
|||||||
impl Program {
|
impl Program {
|
||||||
/// Evaluate the program, returning either an error or a string containing everything
|
/// Evaluate the program, returning either an error or a string containing everything
|
||||||
/// the program printed out.
|
/// the program printed out.
|
||||||
///
|
///
|
||||||
/// The print outs will be newline separated, with one print out per line.
|
/// The print outs will be newline separated, with one print out per line.
|
||||||
pub fn eval(&self) -> Result<String, EvalError> {
|
pub fn eval(&self) -> Result<String, EvalError> {
|
||||||
let mut env = EvalEnvironment::empty();
|
let mut env = EvalEnvironment::empty();
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ use std::collections::HashSet;
|
|||||||
|
|
||||||
impl Program {
|
impl Program {
|
||||||
/// Get the complete list of strings used within the program.
|
/// Get the complete list of strings used within the program.
|
||||||
///
|
///
|
||||||
/// For the purposes of this function, strings are the variables used in
|
/// For the purposes of this function, strings are the variables used in
|
||||||
/// `print` statements.
|
/// `print` statements.
|
||||||
pub fn strings(&self) -> HashSet<ArcIntern<String>> {
|
pub fn strings(&self) -> HashSet<ArcIntern<String>> {
|
||||||
|
|||||||
34
src/lib.rs
34
src/lib.rs
@@ -1,41 +1,41 @@
|
|||||||
//! # NGR (No Good Reason) Compiler
|
//! # NGR (No Good Reason) Compiler
|
||||||
//!
|
//!
|
||||||
//! This is the top-level module for the NGR compiler; a compiler written
|
//! This is the top-level module for the NGR compiler; a compiler written
|
||||||
//! in Rust for no good reason. I may eventually try to turn this into a
|
//! in Rust for no good reason. I may eventually try to turn this into a
|
||||||
//! basic guide for writing compilers, but for now it's a fairly silly
|
//! basic guide for writing compilers, but for now it's a fairly silly
|
||||||
//! (although complete) language and implementation, featuring:
|
//! (although complete) language and implementation, featuring:
|
||||||
//!
|
//!
|
||||||
//! * Variable binding with basic arithmetic operators.
|
//! * Variable binding with basic arithmetic operators.
|
||||||
//! * The ability to print variable values.
|
//! * The ability to print variable values.
|
||||||
//!
|
//!
|
||||||
//! I'll be extending this list into the future, with the eventual goal of
|
//! I'll be extending this list into the future, with the eventual goal of
|
||||||
//! being able to implement basic programming tasks with it. For example,
|
//! being able to implement basic programming tasks with it. For example,
|
||||||
//! I have a goal of eventually writing reasonably-clear
|
//! I have a goal of eventually writing reasonably-clear
|
||||||
//! [Advent of Code](https://adventofcode.com/) implementations with it.
|
//! [Advent of Code](https://adventofcode.com/) implementations with it.
|
||||||
//!
|
//!
|
||||||
//! Users of this as a library will want to choose their adventure based
|
//! Users of this as a library will want to choose their adventure based
|
||||||
//! on how much they want to customize their experience; I've defaulted
|
//! on how much they want to customize their experience; I've defaulted
|
||||||
//! to providing the ability to see internals, rather than masking them,
|
//! to providing the ability to see internals, rather than masking them,
|
||||||
//! so folks can play with things as they see fit.
|
//! so folks can play with things as they see fit.
|
||||||
//!
|
//!
|
||||||
//! ## Easy Mode - Just Running a REPL or Compiler
|
//! ## Easy Mode - Just Running a REPL or Compiler
|
||||||
//!
|
//!
|
||||||
//! For easiest use, you will want to use either the [`Compiler`] object
|
//! For easiest use, you will want to use either the [`Compiler`] object
|
||||||
//! or the [`REPL`] object.
|
//! or the [`REPL`] object.
|
||||||
//!
|
//!
|
||||||
//! As you might expect, the [`Compiler`] object builds a compiler, which
|
//! As you might expect, the [`Compiler`] object builds a compiler, which
|
||||||
//! can be re-used to compile as many files as you'd like. Right now,
|
//! can be re-used to compile as many files as you'd like. Right now,
|
||||||
//! that's all it does. (TODO: Add a linker function to it.)
|
//! that's all it does. (TODO: Add a linker function to it.)
|
||||||
//!
|
//!
|
||||||
//! The [`REPL`] object implements the core of what you'll need to
|
//! The [`REPL`] object implements the core of what you'll need to
|
||||||
//! implement a just-in-time compiled read-eval-print loop. It will
|
//! implement a just-in-time compiled read-eval-print loop. It will
|
||||||
//! maintain variable state and make sure that variables are linked
|
//! maintain variable state and make sure that variables are linked
|
||||||
//! appropriately as the loop progresses.
|
//! appropriately as the loop progresses.
|
||||||
//!
|
//!
|
||||||
//! ## Hard Mode - Looking at the individual passes
|
//! ## Hard Mode - Looking at the individual passes
|
||||||
//!
|
//!
|
||||||
//! This compiler is broken into three core parts:
|
//! This compiler is broken into three core parts:
|
||||||
//!
|
//!
|
||||||
//! 1. The front-end / syntax engine. This portion of the compiler is
|
//! 1. The front-end / syntax engine. This portion of the compiler is
|
||||||
//! responsible for turning basic strings (or files) into a machine-
|
//! responsible for turning basic strings (or files) into a machine-
|
||||||
//! friendly abstract syntax tree. See the [`syntax`] module for
|
//! friendly abstract syntax tree. See the [`syntax`] module for
|
||||||
@@ -49,22 +49,22 @@
|
|||||||
//! helps with either compiling them via JIT or statically compiling
|
//! helps with either compiling them via JIT or statically compiling
|
||||||
//! them into a file. The [`backend`] module also contains information
|
//! them into a file. The [`backend`] module also contains information
|
||||||
//! about the runtime functions made available to the user.
|
//! about the runtime functions made available to the user.
|
||||||
//!
|
//!
|
||||||
//! ## Testing
|
//! ## Testing
|
||||||
//!
|
//!
|
||||||
//! Testing is a key focus of this effort. To that end, both the syntax
|
//! Testing is a key focus of this effort. To that end, both the syntax
|
||||||
//! tree used in the syntax module and the IR used in the middle of the
|
//! tree used in the syntax module and the IR used in the middle of the
|
||||||
//! compiler both implement `Arbitrary`, and are subject to property-based
|
//! compiler both implement `Arbitrary`, and are subject to property-based
|
||||||
//! testing to make sure that various passes work properly.
|
//! testing to make sure that various passes work properly.
|
||||||
//!
|
//!
|
||||||
//! In addition, to support basic equivalence testing, we include support
|
//! In addition, to support basic equivalence testing, we include support
|
||||||
//! for evaluating all expressions. The [`eval`] module provides some
|
//! for evaluating all expressions. The [`eval`] module provides some
|
||||||
//! utility support for this work.
|
//! utility support for this work.
|
||||||
//!
|
//!
|
||||||
pub mod syntax;
|
|
||||||
pub mod ir;
|
|
||||||
pub mod backend;
|
pub mod backend;
|
||||||
pub mod eval;
|
pub mod eval;
|
||||||
|
pub mod ir;
|
||||||
|
pub mod syntax;
|
||||||
|
|
||||||
/// Implementation module for the high-level compiler.
|
/// Implementation module for the high-level compiler.
|
||||||
mod compiler;
|
mod compiler;
|
||||||
|
|||||||
14
src/repl.rs
14
src/repl.rs
@@ -10,11 +10,11 @@ use pretty::termcolor::{ColorChoice, StandardStream};
|
|||||||
use std::collections::HashMap;
|
use std::collections::HashMap;
|
||||||
|
|
||||||
/// A high-level REPL helper for NGR.
|
/// A high-level REPL helper for NGR.
|
||||||
///
|
///
|
||||||
/// This object holds most of the state required to implement some
|
/// This object holds most of the state required to implement some
|
||||||
/// form of interactive compiler for NGR; all you need to do is provide
|
/// form of interactive compiler for NGR; all you need to do is provide
|
||||||
/// the actual user IO.
|
/// the actual user IO.
|
||||||
///
|
///
|
||||||
/// For most console-based used cases, the [`Default`] implementation
|
/// For most console-based used cases, the [`Default`] implementation
|
||||||
/// should be sufficient; it prints any warnings or errors to `stdout`,
|
/// should be sufficient; it prints any warnings or errors to `stdout`,
|
||||||
/// using a default color scheme that should work based on the terminal
|
/// using a default color scheme that should work based on the terminal
|
||||||
@@ -62,7 +62,7 @@ impl From<REPLError> for Diagnostic<usize> {
|
|||||||
|
|
||||||
impl REPL {
|
impl REPL {
|
||||||
/// Construct a new REPL helper, using the given stream implementation and console configuration.
|
/// Construct a new REPL helper, using the given stream implementation and console configuration.
|
||||||
///
|
///
|
||||||
/// For most users, the [`Default::default`] implementation will be sufficient;
|
/// For most users, the [`Default::default`] implementation will be sufficient;
|
||||||
/// it will use `stdout` and a default console configuration. But if you need to
|
/// it will use `stdout` and a default console configuration. But if you need to
|
||||||
/// be more specific, this will help you provide more guidance to the REPL as it
|
/// be more specific, this will help you provide more guidance to the REPL as it
|
||||||
@@ -79,7 +79,7 @@ impl REPL {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Emit a diagnostic to the configured console.
|
/// Emit a diagnostic to the configured console.
|
||||||
///
|
///
|
||||||
/// This is just a convenience function; there's a lot of boilerplate in printing
|
/// This is just a convenience function; there's a lot of boilerplate in printing
|
||||||
/// diagnostics, and it was nice to pull it out into its own function.
|
/// diagnostics, and it was nice to pull it out into its own function.
|
||||||
fn emit_diagnostic(
|
fn emit_diagnostic(
|
||||||
@@ -95,13 +95,13 @@ impl REPL {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Process a line of input, printing any problems or the results.
|
/// Process a line of input, printing any problems or the results.
|
||||||
///
|
///
|
||||||
/// The line number argument is just for a modicum of source information, to
|
/// The line number argument is just for a modicum of source information, to
|
||||||
/// provide to the user if some parsing or validation step fails. It can be
|
/// provide to the user if some parsing or validation step fails. It can be
|
||||||
/// changed to be any value you like that provides some insight into what
|
/// changed to be any value you like that provides some insight into what
|
||||||
/// failed, although it is probably a good idea for it to be different for
|
/// failed, although it is probably a good idea for it to be different for
|
||||||
/// every invocation of this function. (Not critical, but a good idea.)
|
/// every invocation of this function. (Not critical, but a good idea.)
|
||||||
///
|
///
|
||||||
/// Any warnings or errors generated in processing this command will be
|
/// Any warnings or errors generated in processing this command will be
|
||||||
/// printed to the configured console. If there are no problems, the
|
/// printed to the configured console. If there are no problems, the
|
||||||
/// command will be compiled and then executed.
|
/// command will be compiled and then executed.
|
||||||
@@ -117,7 +117,7 @@ impl REPL {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// The internal implementation, with a handy `Result` type.
|
/// The internal implementation, with a handy `Result` type.
|
||||||
///
|
///
|
||||||
/// All information from the documentation of `REPL::process_input` applies here,
|
/// All information from the documentation of `REPL::process_input` applies here,
|
||||||
/// as well; this is the internal implementation of that function, which is
|
/// as well; this is the internal implementation of that function, which is
|
||||||
/// differentiated by returning a `Result` type that is hidden from the user
|
/// differentiated by returning a `Result` type that is hidden from the user
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
//! NGR Parsing: Reading input, turning it into sense (or errors).
|
//! NGR Parsing: Reading input, turning it into sense (or errors).
|
||||||
//!
|
//!
|
||||||
//! This module implement the front end of the compiler, which is responsible for
|
//! This module implement the front end of the compiler, which is responsible for
|
||||||
//! reading in NGR syntax as a string, turning it into a series of reasonable Rust
|
//! reading in NGR syntax as a string, turning it into a series of reasonable Rust
|
||||||
//! structures for us to manipulate, and doing some validation while it's at it.
|
//! structures for us to manipulate, and doing some validation while it's at it.
|
||||||
//!
|
//!
|
||||||
//! The core flow for this work is:
|
//! The core flow for this work is:
|
||||||
//!
|
//!
|
||||||
//! * Turning the string into a series of language-specific [`Token`]s.
|
//! * Turning the string into a series of language-specific [`Token`]s.
|
||||||
//! * Taking those tokens, and computing a basic syntax tree from them,
|
//! * Taking those tokens, and computing a basic syntax tree from them,
|
||||||
//! using our parser ([`ProgramParser`] or [`StatementParser`], generated
|
//! using our parser ([`ProgramParser`] or [`StatementParser`], generated
|
||||||
@@ -15,12 +15,12 @@
|
|||||||
//! * Simplifying the tree we have parsed, using the [`simplify`] module,
|
//! * Simplifying the tree we have parsed, using the [`simplify`] module,
|
||||||
//! into something that's more easily turned into our [compiler internal
|
//! into something that's more easily turned into our [compiler internal
|
||||||
//! representation](super::ir).
|
//! representation](super::ir).
|
||||||
//!
|
//!
|
||||||
//! In addition to all of this, we make sure that the structures defined in this
|
//! In addition to all of this, we make sure that the structures defined in this
|
||||||
//! module are all:
|
//! module are all:
|
||||||
//!
|
//!
|
||||||
//! * Instances of [`Pretty`](::pretty::Pretty), so that you can print stuff back
|
//! * Instances of [`Pretty`](::pretty::Pretty), so that you can print stuff back
|
||||||
//! out that can be read by a human.
|
//! out that can be read by a human.
|
||||||
//! * Instances of [`Arbitrary`](proptest::prelude::Arbitrary), so they can be
|
//! * Instances of [`Arbitrary`](proptest::prelude::Arbitrary), so they can be
|
||||||
//! used in `proptest`-based property testing. There are built-in tests in
|
//! used in `proptest`-based property testing. There are built-in tests in
|
||||||
//! the library, for example, to make sure that the pretty-printing round-trips.
|
//! the library, for example, to make sure that the pretty-printing round-trips.
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
use codespan_reporting::diagnostic::{Diagnostic, Label};
|
use codespan_reporting::diagnostic::{Diagnostic, Label};
|
||||||
|
|
||||||
/// A source location, for use in pointing users towards warnings and errors.
|
/// A source location, for use in pointing users towards warnings and errors.
|
||||||
///
|
///
|
||||||
/// Internally, locations are very tied to the `codespan_reporting` library,
|
/// Internally, locations are very tied to the `codespan_reporting` library,
|
||||||
/// and the primary use of them is to serve as anchors within that library.
|
/// and the primary use of them is to serve as anchors within that library.
|
||||||
#[derive(Clone, Debug, Eq, PartialEq)]
|
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||||
@@ -13,7 +13,7 @@ pub struct Location {
|
|||||||
impl Location {
|
impl Location {
|
||||||
/// Generate a new `Location` from a file index and an offset from the
|
/// Generate a new `Location` from a file index and an offset from the
|
||||||
/// start of the file.
|
/// start of the file.
|
||||||
///
|
///
|
||||||
/// The file index is based on the file database being used. See the
|
/// The file index is based on the file database being used. See the
|
||||||
/// `codespan_reporting::files::SimpleFiles::add` function, which is
|
/// `codespan_reporting::files::SimpleFiles::add` function, which is
|
||||||
/// normally where we get this index.
|
/// normally where we get this index.
|
||||||
@@ -22,7 +22,7 @@ impl Location {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Generate a `Location` for a completely manufactured bit of code.
|
/// Generate a `Location` for a completely manufactured bit of code.
|
||||||
///
|
///
|
||||||
/// Ideally, this is used only in testing, as any code we generate as
|
/// Ideally, this is used only in testing, as any code we generate as
|
||||||
/// part of the compiler should, theoretically, be tied to some actual
|
/// part of the compiler should, theoretically, be tied to some actual
|
||||||
/// location in the source code. That being said, this can be used in
|
/// location in the source code. That being said, this can be used in
|
||||||
@@ -36,10 +36,10 @@ impl Location {
|
|||||||
|
|
||||||
/// Generate a primary label for a [`Diagnostic`], based on this source
|
/// Generate a primary label for a [`Diagnostic`], based on this source
|
||||||
/// location.
|
/// location.
|
||||||
///
|
///
|
||||||
/// Note, this is just the [`Label`], you'll want to fill in the [`Diagnostic`]
|
/// Note, this is just the [`Label`], you'll want to fill in the [`Diagnostic`]
|
||||||
/// with a lot more information.
|
/// with a lot more information.
|
||||||
///
|
///
|
||||||
/// Primary labels are the things that are they key cause of the message.
|
/// Primary labels are the things that are they key cause of the message.
|
||||||
/// If, for example, it was an error to bind a variable named "x", and
|
/// If, for example, it was an error to bind a variable named "x", and
|
||||||
/// then have another binding of a variable named "x", the second one
|
/// then have another binding of a variable named "x", the second one
|
||||||
@@ -52,10 +52,10 @@ impl Location {
|
|||||||
|
|
||||||
/// Generate a secondary label for a [`Diagnostic`], based on this source
|
/// Generate a secondary label for a [`Diagnostic`], based on this source
|
||||||
/// location.
|
/// location.
|
||||||
///
|
///
|
||||||
/// Note, this is just the [`Label`], you'll want to fill in the [`Diagnostic`]
|
/// Note, this is just the [`Label`], you'll want to fill in the [`Diagnostic`]
|
||||||
/// with a lot more information.
|
/// with a lot more information.
|
||||||
///
|
///
|
||||||
/// Secondary labels are the things that are involved in the message, but
|
/// Secondary labels are the things that are involved in the message, but
|
||||||
/// aren't necessarily a problem in and of themselves. If, for example, it
|
/// aren't necessarily a problem in and of themselves. If, for example, it
|
||||||
/// was an error to bind a variable named "x", and then have another binding
|
/// was an error to bind a variable named "x", and then have another binding
|
||||||
@@ -63,20 +63,20 @@ impl Location {
|
|||||||
/// label (because that's where the error actually happened), but you'd
|
/// label (because that's where the error actually happened), but you'd
|
||||||
/// probably want to make the first location the secondary label to help
|
/// probably want to make the first location the secondary label to help
|
||||||
/// users find it.
|
/// users find it.
|
||||||
pub fn secondary_label(&self) -> Label<usize> {
|
pub fn secondary_label(&self) -> Label<usize> {
|
||||||
Label::secondary(self.file_idx, self.offset..self.offset)
|
Label::secondary(self.file_idx, self.offset..self.offset)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Given this location and another, generate a primary label that
|
/// Given this location and another, generate a primary label that
|
||||||
/// specifies the area between those two locations.
|
/// specifies the area between those two locations.
|
||||||
///
|
///
|
||||||
/// See [`Self::primary_label`] for some discussion of primary versus
|
/// See [`Self::primary_label`] for some discussion of primary versus
|
||||||
/// secondary labels. If the two locations are the same, this method does
|
/// secondary labels. If the two locations are the same, this method does
|
||||||
/// the exact same thing as [`Self::primary_label`]. If this item was
|
/// the exact same thing as [`Self::primary_label`]. If this item was
|
||||||
/// generated by [`Self::manufactured`], it will act as if you'd called
|
/// generated by [`Self::manufactured`], it will act as if you'd called
|
||||||
/// `primary_label` on the argument. Otherwise, it will generate the obvious
|
/// `primary_label` on the argument. Otherwise, it will generate the obvious
|
||||||
/// span.
|
/// span.
|
||||||
///
|
///
|
||||||
/// This function will return `None` only in the case that you provide
|
/// This function will return `None` only in the case that you provide
|
||||||
/// labels from two different files, which it cannot sensibly handle.
|
/// labels from two different files, which it cannot sensibly handle.
|
||||||
pub fn range_label(&self, end: &Location) -> Option<Label<usize>> {
|
pub fn range_label(&self, end: &Location) -> Option<Label<usize>> {
|
||||||
@@ -96,7 +96,7 @@ impl Location {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Return an error diagnostic centered at this location.
|
/// Return an error diagnostic centered at this location.
|
||||||
///
|
///
|
||||||
/// Note that this [`Diagnostic`] will have no information associated with
|
/// Note that this [`Diagnostic`] will have no information associated with
|
||||||
/// it other than that (a) there is an error, and (b) that the error is at
|
/// it other than that (a) there is an error, and (b) that the error is at
|
||||||
/// this particular location. You'll need to extend it with actually useful
|
/// this particular location. You'll need to extend it with actually useful
|
||||||
@@ -109,7 +109,7 @@ impl Location {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Return an error diagnostic centered at this location, with the given message.
|
/// Return an error diagnostic centered at this location, with the given message.
|
||||||
///
|
///
|
||||||
/// This is much more useful than [`Self::error`], because it actually provides
|
/// This is much more useful than [`Self::error`], because it actually provides
|
||||||
/// the user with some guidance. That being said, you still might want to add
|
/// the user with some guidance. That being said, you still might want to add
|
||||||
/// even more information to ut, using [`Diagnostic::with_labels`],
|
/// even more information to ut, using [`Diagnostic::with_labels`],
|
||||||
|
|||||||
@@ -6,28 +6,28 @@ use thiserror::Error;
|
|||||||
|
|
||||||
/// A single token of the input stream; used to help the parsing go down
|
/// A single token of the input stream; used to help the parsing go down
|
||||||
/// more easily.
|
/// more easily.
|
||||||
///
|
///
|
||||||
/// The key way to generate this structure is via the [`Logos`] trait.
|
/// The key way to generate this structure is via the [`Logos`] trait.
|
||||||
/// See the [`logos`] documentation for more information; we use the
|
/// See the [`logos`] documentation for more information; we use the
|
||||||
/// [`Token::lexer`] function internally.
|
/// [`Token::lexer`] function internally.
|
||||||
///
|
///
|
||||||
/// The first step in the compilation process is turning the raw string
|
/// The first step in the compilation process is turning the raw string
|
||||||
/// data (in UTF-8, which is its own joy) in to a sequence of more sensible
|
/// data (in UTF-8, which is its own joy) in to a sequence of more sensible
|
||||||
/// tokens. Here, for example, we turn "x=5" into three tokens: a
|
/// tokens. Here, for example, we turn "x=5" into three tokens: a
|
||||||
/// [`Token::Variable`] for "x", a [`Token::Equals`] for the "=", and
|
/// [`Token::Variable`] for "x", a [`Token::Equals`] for the "=", and
|
||||||
/// then a [`Token::Number`] for the "5". Later on, we'll worry about
|
/// then a [`Token::Number`] for the "5". Later on, we'll worry about
|
||||||
/// making sense of those three tokens.
|
/// making sense of those three tokens.
|
||||||
///
|
///
|
||||||
/// For now, our list of tokens is relatively straightforward. We'll
|
/// For now, our list of tokens is relatively straightforward. We'll
|
||||||
/// need/want to extend these later.
|
/// need/want to extend these later.
|
||||||
///
|
///
|
||||||
/// The [`std::fmt::Display`] implementation for [`Token`] should
|
/// The [`std::fmt::Display`] implementation for [`Token`] should
|
||||||
/// round-trip; if you lex a string generated with the [`std::fmt::Display`]
|
/// round-trip; if you lex a string generated with the [`std::fmt::Display`]
|
||||||
/// trait, you should get back the exact same token.
|
/// trait, you should get back the exact same token.
|
||||||
#[derive(Logos, Clone, Debug, PartialEq, Eq)]
|
#[derive(Logos, Clone, Debug, PartialEq, Eq)]
|
||||||
pub enum Token {
|
pub enum Token {
|
||||||
// Our first set of tokens are simple characters that we're
|
// Our first set of tokens are simple characters that we're
|
||||||
// going to use to structure NGR programs.
|
// going to use to structure NGR programs.
|
||||||
#[token("=")]
|
#[token("=")]
|
||||||
Equals,
|
Equals,
|
||||||
|
|
||||||
@@ -74,7 +74,7 @@ pub enum Token {
|
|||||||
#[regex(r"[ \t\r\n\f]+", logos::skip)]
|
#[regex(r"[ \t\r\n\f]+", logos::skip)]
|
||||||
// this is an extremely simple version of comments, just line
|
// this is an extremely simple version of comments, just line
|
||||||
// comments. More complicated /* */ comments can be harder to
|
// comments. More complicated /* */ comments can be harder to
|
||||||
// implement, and didn't seem worth it at the time.
|
// implement, and didn't seem worth it at the time.
|
||||||
#[regex(r"//.*", logos::skip)]
|
#[regex(r"//.*", logos::skip)]
|
||||||
/// This token represents that some core error happened in lexing;
|
/// This token represents that some core error happened in lexing;
|
||||||
/// possibly that something didn't match anything at all.
|
/// possibly that something didn't match anything at all.
|
||||||
|
|||||||
Reference in New Issue
Block a user