Release v0.11.0

There were ~248 commits between v0.10.0 and this release. A lot of bugfixes, performance improvements, and new features. This release is the closest lde has ever been to a stable release.

Preface

Briefly, I will say, since the release of 0.10.0, I have had a change of mindset that has led me to limiting the amount of llm usage when it comes to documentation. Hence, going forward, this, and any future blog post alongside all documentation, will be entirely written and handled manually (excluding autogenerated content like fix lists).

I am not against llms. But I think lde needs a proper voice to explain why it is special.

Error Rendering

All too often do you encounter errors in your tests, code, etc.

Now, just as tests render their output neatly, failures in your code executed by lde render their output printing out the source code that caused the error.

LuaRocks Improvements

LuaRocks support has been greatly expanded, with custom luarocks backend support, and fixes to ensure lukefile packages work flawlessly.

Custom LuaRocks Backends

These “backends” are equivalent to lde’s build.lua, except they have access to all of luarocks’ internals, so we must polyfill them in order for them to function correctly.

We do this via running them in a secure sandbox (the same way lde runs your scripts) with some of those modules polyfilled using lde’s filesystem.

This unblocks support for packages like toml-edit

Lukefile Fixes

Environment variables were not truly passed to intermediate programs, so for example, command type luarocks packages, ie lukefiles, were not expanding variables passed to luarocks packages.

Fixing this unblocks support for a lot more packages, like lyaml.

Built-in Teal Support

Teal is a statically typed dialect of Lua, comparable to how TypeScript is a statically typed dialect of JavaScript.

Now you can write .tl files and use them alongside your .lua files and they will work seamlessly with lde!

Read more about teal support in the docs.

function greet(name: string)
    print("Hello, " .. name)
end

greet("lde")

lde ./example.tl
# Hello, lde

This works by running the official teal compiler via lde’s rich luarocks support on your files ahead of time before running lua.

Built-in MoonScript Support

MoonScript is a dynamic language atop Lua aiming to provide a more streamlined experience, as CoffeeScript was to JavaScript.

As with Teal support, you can now write .moon files and use them in your projects seamlessly.

class Thing
  name: "unknown"

class Person extends Thing
  say_name: => print "Hello, I am #{@name}!"

with Person!
  .name = "MoonScript"
  \say_name!

lde new --language moonscript --name foo --type blank ./foo
# <write the code snippet above into init.moon>
cd ./foo && lde run
# Hello, I am MoonScript!

This runs the official moonscript compiler, so it will always remain up to date.

Interactive Init

Now lde new and lde init give you an interactive prompt to set up your project.

New Documentation

The website has been overhauled in how it looks.

But the biggest change is that all docs have been entirely refreshed and rewritten. There are no longer any outdated docs, nor any LLM slopped docs. Blog posts will not contain any llm written text from this version forward.

Additionally, they’re now much more llm friendly, you can easily copy pages as .md files, or feed your llm a comprehensive md file for it to understand lde by telling it:

Read https://lde.sh/llms.txt and set up a new project.

Since they are human written and intended to be compact, they’re small on tokens too.

JIT Debugging

With LuaJIT, for performance, your goal is to ensure your code compiles to native code as often as possible.

This is what it means for your code to “JIT” compile. In some circumstances, your code may do something LuaJIT does not permit to be JIT compiled, or goes over a certain limit for it to do so.

For this reason, the --jit flag now exists to monitor when these occur and why.

It will print out to you as your program runs when a fault occurs, and at the end, a summary with source code showing where the failure occurred will be printed.

In this case, the user tries to create a function at runtime. This is a big no-no for JIT compilation

This is incredibly helpful to ensure your code remains as fast, and as idiomatic LuaJIT as possible.

You can read more about this here

Registry Searching

Searching the registry via the website is easy. But now you can do it on your terminal with lde search!

You can read more about it here

Hotreloading

This is a new alternative to --watch that allows you to reload individual files without restarting your entire program.

It is useful for things like hosting servers, or keeping a file handle, you don’t want to lose from a full restart.

It will patch the results of require() outputs at runtime for your entrypoint to use.

API

With hotreloading, it additionally adds an api to the package library, package.hot, when --hot is enabled.

if package.hot then
    package.hot.accept(|m| -> print("module " .. m .. " was reloaded!"))
end

You can read more about hot reloading here.

Test Coverage

A new flag has been added to the lde test runner, via lde test --coverage.

This runs your test suite, and reports the % of executable lines that are covered by your tests.

It is a useful way to try and make your code as bug-resilient as possible.

Read more about it here.

Shell Completions

Shell completions are now provided by lde completion.

lde completion <bash|zsh|fish>

Add this to your shell config to enable completions, for example, bash:

eval "$(lde completion bash)"

Git Shorthands

Shorthands have been added for easier git integration across commands.

Example

lde add foo --git https://github.com/bar/baz
# Is equivalent to:
lde add gh:foo@bar/baz

This also works for github, gitlab and codeberg.

Build Timings

Now you can get a timings report for what takes the longest to build in your programs with lde <sync/run/compile> --timings.

Read more about it here.

Bloat Detection

When using lde, chances are you’ll be distributing your code in a binary or lua file with lde compile / lde bundle.

The new lde bloat command will help you determine what makes up this bundle, so you can do things like eliminate dependencies, or look at files to shrink.

This is an interactive TUI you can walk around to expand dependencies to look at individual files contributing to your bundle size.

Read more about it here.

Cross Compilation

You can now cross compile to other platforms, granted you have the tooling on your OS.

Do this with lde compile --target <target> where target is one of the triples here:

• linux-x86-64
• linux-aarch64
• windows-x86-64
• windows-aarch64
• macos-x86-64
• macos-aarch64
• android-aarch64

Read more about this feature here.

It is recommended you use clang rather than gcc, as its cross compilation story is slightly better.

Quiet Builds

Builds are now quiet by default. This leads to less flooding in your output, and errors will be printed out if encountered, as well as with a full log written to your tmp directory.

If you wish to see the full output of a build as before, you can use the --verbose flag.

Parallel Builds

Builds for luarocks packages would already be downloaded and built in parallel.

This now also applies to lde’s build.lua scripts and packages. This made lde’s build time significantly shorter, as it became as long as the slowest dependency, rather than the sum of all dependencies.

Standardized Errors

Errors have been centralized into one system that prints neatly into your console and returns a nonzero error code.

While before, certain things would cause hard crashes in lde, some stuff would throw an error(), others would print with ansi coloring. It was pretty inconsistent.

Hard Crashes

Additionally, actual hard crashes now print a stack trace and tell you to report them as a real bug!

Registry Improvements

The quality of life of using the lde registry has been significantly improved.

Automation

Publishing a PR to add a package to the registry will automatically be reviewed by a bot, to ensure the schema is correct, and upon merging by a moderator, any future PRs to update the package to add versions or change metadata will automatically be approved.

Things like last updated, ownership metadata, will be recorded and update immediately on the registry as seen below:

This allows for the clean-room openness of GitHub with the seamless usage of a self hosted registry.

Namespaces

Namespaces are supported and automated. Simply request a namespace, with the button on the registry page which auto-fills an issue template for you, and a moderator will approve or deny it.

New Distributions

Musl

Musl builds are now available for Linux, both x86-64 and ARM64.

These are useful for using lde inside of constrained environments, like with Alpine Linux.

The typical install workflow will detect the libc of your operating system, so you can follow the standard download steps here

Docker

Docker images now exist and are published to the GitHub Container Registry (GHCR) on every nightly and release build.

There are three variants:

Where <version> is the version of lde you want to install, or nightly for the latest nightly build.

Install using docker pull ghcr.io/lde-org/lde:<name>

LuaRocks

Luarocks was technically supported since 0.8.1 but it was left unmaintained. Now it is automatically updated in CI, for both nightly and fixed releases.

Install lde with a simple command:

luarocks install lde

Fixes

• errors: render syntax and load errors with a source snippet instead of the raw loader frame
• errors: show the real source path for parse errors in a package entry point
• errors: name the broken module when a require() fails to compile
• errors: thrown table errors render instead of crashing the renderer
• errors: attribute errors in nested test files to the file that raised them
• errors: show the compiler’s own position for Teal and MoonScript errors
• errors: keep the caret line directly beneath the failing line
• errors: report a deleted working directory cleanly, with a hint
• errors: report the missing file for lde ./nope.lua instead of Unknown command
• test: fail when a filter matches nothing, or when no tests exist at all
• test: refresh tests/ helpers when their source changes (they were served stale)
• remove: allow removing dev dependencies
• add: parse lde add --dev foo the same as lde add foo --dev
• add: create the dependencies object when the manifest has none
• add: stop discarding --path and --git
• update: exit non-zero for an unknown dependency
• update: leave a dependency pinned by commit in lde.json alone
• lockfile: invalidate when lde.json changes, so a switched source is not installed
• lockfile: record registry dependencies with their git source, not a bare commit
• install: install single-file rocks under their name (fixes the semver rock)
• install: rebuild into a clean directory, so removed files do not survive
• install: recover from cache errors caused by transient download failures
• install: keep the URL cache for --offline instead of discarding it
• install: recover cleanly from a manually deleted target/
• hot: invalidate a Teal source’s compiled output, which kept reloading stale code
• windows: run lde.json scripts under sh instead of cmd
• windows: do not escape strings passed to the command shell
• windows: do not escape the path in the lde x wrapper
• windows: always create the output directory
• macos: use the proper monotonic clock
• repl: stop lazily requiring modules, which broke the REPL
• profiler: respect quiet mode
• bloat: normalize reported paths
• sea: allow lde compile to replace its own binary
• ansi: clear wrapped progress lines by terminal width
• luarocks: allow rockspecs with a . in the package name
• luarocks: allow 3.0+ rockspecs to omit the build section
• luarocks: pass environment variables to command builds even when the rockspec does not use them
• luarocks: avoid busted looping on itself
• bundle: do not embed test files
• x: pass -- through to the ldx wrapper
• general: discover the git repository from nested folders
• general: make lde-build:copy() atomic

Ending Note

This took a long time to write. I’m tired. I also broke my promise of this version not taking so long to be released from the last ending note, so that’s on me.

At this point, I don’t see much holding lde back from a 1.0 release. So I will hopefully see you there soon :)