package.el
, which installs packages from the official Emacs Lisp Package Archive, named GNU ELPA.
GNU ELPA hosts a selection of packages, but most are available on MELPA, which is an unofficial package archive that implements the ELPA specification. To use MELPA, it has to be installed by adding it to the list of package.el
package archives.
The built-in package manager installs packages through the package-install
function. For example, to install the “evil-commentary” package from MELPA, call package-install
inside Emacs:
M-x
package-install
<RET>
evil-commentary
<RET>
Straight.el is an alternative package manager that installs packages through Git checkouts instead of downloading tarballs from one of the package archives. Doing so allows installing forked packages, altering local package checkouts, and locking packages to exact versions for reproducable setups.
The Getting started section in the straight.el README provides the bootstrap code to place inside ~/.emacs.d/init.el
in order to install it:
;; Install straight.el (defvar bootstrap-version) (let ((bootstrap-file (expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory)) (bootstrap-version 5)) (unless (file-exists-p bootstrap-file) (with-current-buffer (url-retrieve-synchronously "https://raw.githubusercontent.com/raxod502/straight.el/develop/install.el" 'silent 'inhibit-cookies) (goto-char (point-max)) (eval-print-last-sexp))) (load bootstrap-file nil 'nomessage))
Straight.el uses package archives like GNU ELPA as registries to find the linked repositories to clone from. Since these are checked automatically, there’s no need to add them to the list of package archives.
While package.el loads all installed packages on startup, straight.el only loads packages that are referenced in the init file. This allows for installing packages temporarily without slowing down Emacs’ startup time on subsequent startups.
To create a truly reproducable setup, disable package.el in favor of straight.el by turning off package-enable-at-startup
. Because this step needs to happen before package.el gets a chance to load packages, it this configuration needs to be set in the early init file:
;; Disable package.el in favor of straight.el (setq package-enable-at-startup nil)
With this configuration set, Emacs will only load the packages installed through straight.el.
To use straight.el to install a package for the current session, execute the straight-use-package
command:
M-x
straight-use-package
<RET>
evil-commentary
<RET>
To continue using the package in future sessions, add the straight-use-package
call to ~/.emacs/init.el
:
(straight-use-package 'evil-commentary)
To update an installed package, execute the straight-pull-package
command:
M-x
straight-pull-package
<RET>
evil-commentary
<RET>
To update the version lockfile, which is used to target the exact version to check out when installing, run straight-freeze-versions
:
M-x
straight-freeze-versions
<RET>
Use-package is a macro to configure and load packages in Emacs configurations. It interfaces with package managers like package.el or straight.el to install packages, but is not a package manager by itself.
For example, when using straight.el without use-package, installing and starting evil-commentary requires installing the package and starting it as two separate steps:
(straight-use-package 'evil-commentary) (evil-commentary-mode)
Combined with use-package, the installation and configuration are unified into a single call to use-package
:
(use-package evil-commentary
:config (evil-commentary-mode))
Aside from keeping configuration files tidy, having package configuration contained within a single call allows for more advanced package setups. For example, packages can be lazy-loaded, keeping their configuration code from executing until the package they configure is needed.
To install use-package with straight.el, use straight-use-package
:
;; Install use-package (straight-use-package 'use-package)
By default, use-package uses package.el to install packages. To use straight.el instead of package.el, pass the :straight
option:
(use-package evil-commentary
:straight t)
To configure use-package to always use straight.el, use use-package
to configure straight.el to turn on straight-use-package-by-default
1:
;; Configure use-package to use straight.el by default (use-package straight :custom (straight-use-package-by-default t))
Now, installing any package using use-package uses straight.el, even when omitting the :straight.el
option.
Having both straight.el and use-package installed and configured to work together, the straight-use-package
function isn’t used anymore. Instead, all packages are installed and configured through use-package.
Use the use-package
macro to load a package. If the package is not installed yet, it is installed automatically:
(use-package evil-commentary)
Use-package provides keywords to add configuration, key bindings and variables. Although there are many more options, some examples include :config
, :init
, :bind
, and :custom
:
:config
and :init
The :config
and :init
configuration keywords define code that’s run right after, or right before a package is loaded, respectively.
For example, call evil-mode
from the :config
keyword to start Evil after loading its package. To turn off evil-want-C-i-jump
right before evil is loaded (instead of adding it to the early init file), configure it in the :init
keyword:
(use-package evil :init (setq evil-want-C-i-jump nil) :config (evil-mode))
:bind
Adds key bindings after a module is loaded. For example, to use consult-buffer
instead of the built-in switch-to-buffer
after loading the consult package, add a binding through the :bind
keyword:
(use-package consult :bind ("C-x b" . consult-buffer)
:custom
Sets customizable variables. The variables set through use-package are not saved in Emacs’ custom file. Instead, all custom variables are expected to be set through use-package. In an example from before, the :custom
keyword is used to set the straight-use-package-by-default
configuration option after loading straight.el:
(use-package straight
:custom (straight-use-package-by-default t))
The resulting ~/.emacs.d/init.el
file installs straight.el and use-package, and configures straight.el as the package manager for use-package to use:
;; Install straight.el (defvar bootstrap-version) (let ((bootstrap-file (expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory)) (bootstrap-version 5)) (unless (file-exists-p bootstrap-file) (with-current-buffer (url-retrieve-synchronously "https://raw.githubusercontent.com/raxod502/straight.el/develop/install.el" 'silent 'inhibit-cookies) (goto-char (point-max)) (eval-print-last-sexp))) (load bootstrap-file nil 'nomessage)) ;; Install use-package (straight-use-package 'use-package) ;; Configure use-package to use straight.el by default (use-package straight :custom (straight-use-package-by-default t))
The ~/.emacs.d/early-init.el
file disables package.el to disable its auto-loading, causing all packages to be loaded through straight.el in the init file:
;; Disable package.el in favor of straight.el (setq package-enable-at-startup nil)
This is the only configuration set in the early init file. All other packages are installed and configured through use-package, which makes sure to load configuration options before packages are loaded, if configured with the :init
keyword.
Calling use-package
would normally install straight.el, but since it’s already installed, the installation is skipped and the configuration is set. Here, the call to use-package
is only used to configure straight.el, by setting the straight-use-package-by-default
option.
--allow-unrelated-histories
command line option.
As an example, there are two repositories that both have a single root commit. Running git log
produces the log for the first repository, as that repository’s directory is the current directory:
git log
commit 733ed008d41505623d6f3b5dbee7d1841a959c34 Author: Alice <alice@example.com> Date: Sun Aug 8 20:12:38 2021 +0200 Add one.txt
To get the logs for the second repository, pass a path to that repository’s .git
directory to the git log
command via the --git-dir
command line option:
git --git-dir=../two/.git log
commit 43d2970b4dafa21477e1a5a86cad45bc5e798696 Author: Bob <bob@example.com> Date: Sun Aug 8 20:12:44 2021 +0200 Add two.txt
To combine the two repositories, first add the second repository as a remote to the first. Then, run git fetch
to fetch its branch information:
git remote add two ../two git fetch two
Then merge, with the remote set up, merge the second repository’s history into the first by using the --allow-unrelated-histories
flag:
git merge two/main --allow-unrelated-histories
Merge made by the 'recursive' strategy. two.txt | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 two.txt
Unlike extracting part of a repository—which rewrites history by definition—combining repositories is done by merging their two histories. This means the commits from the second repository are appended to the ones from the first, retaining history:
git log
commit 0bd5727a597077b5a5db9590b3f6aaf70eb10b60 Merge: 733ed00 43d2970 Author: Alice <alice@example.com> Date: Sun Aug 8 20:13:01 2021 +0200 Merge remote-tracking branch 'two/main' into main commit 43d2970b4dafa21477e1a5a86cad45bc5e798696 Author: Bob <bob@example.com> Date: Sun Aug 8 20:12:44 2021 +0200 Add two.txt commit 733ed008d41505623d6f3b5dbee7d1841a959c34 Author: Alice <alice@example.com> Date: Sun Aug 8 20:12:38 2021 +0200 Add one.txt
https://git-scm.com/docs/git-merge:
By default,
git merge
command refuses to merge histories that do not share a common ancestor. [The--allow-unrelated-histories
] option can be used to override this safety when merging histories of two projects that started their lives independently.
asciidoctor \ --backend docbook \ --out-file - \ hello.adoc | \ pandoc \ --from docbook \ --wrap=preserve \ --standalone \ --output hello.org
This command pipes two commands together to produce the desired result:
asciidoctor --backend docbook --out-file - hello.adoc
hello.adoc
file and convert it to Docbook. Pass -
as the --out-file
to return the output to standard outputpandoc --from-docbook --wrap=preserve --standalone --output hello.org
Given an AsciiDoc file:
= Hello, World! Alice 2021-08-06 Hello, World! [source,ruby] ---- puts "Hello, World!" ----
The command above produces an Org file including the document headers, contents, and code blocks:
#+title: Hello, World! #+author: Alice #+date: 2021-08-06 Hello, World! #+begin_src ruby puts Hello, World! #+end_src
https://lists.gnu.org/archive/html/emacs-orgmode/2018-02/msg00295.html:
I have a large document (a book) written in AsciiDoc, and I’ve been thinking of converting it to org-mode, which I find eminently more readable. The method I’ve come up with is:
- AsciiDoc -> Docbook using asciidoc or asciidoctor
- Docbook -> org using pandoc
https://pandoc.org/MANUAL.html#using-pandoc:
By default, pandoc produces a document fragment. To produce a standalone document (e.g. a valid HTML file including
<head>
and<body>
), use the-s
or--standalone
flag:pandoc -s -o output.html input.txt
For example, when writing an article about Ruby, it might be useful to show how to start a web server and display the output printed to the command line:
ruby server.rb
Server running on http://localhost:4567...
In the Org file, that code block might look like this:
Finally, start the server by running =ruby server.rb=: #+begin_src shell ruby server.rb #+end_src
This example includes the command to start the server (ruby server.rb
) and sets the :exports
header to export both the input command and the results of running it.
However, because the server is a long-running process, exporting this document causes the exporter to hang. It’s waiting for a result from the command which never comes because the server remains open to connections until it’s quit.1
To start a long-running process and quit it after some time has passed on the command line, use the timeout
utility to automatically kill the process after a predefined delay:
timeout 1 ruby server.rb; echo "Done!"
Server running on http://localhost:4567... Done!
To translate this to an Org code block, use a :prologue
and :epilogue
to add the timeout without showing it in the exported file. The :prologue
prefixes the command with the call to the timeout
utility and the :epilogue
appends :
to the command to make sure the block returns a zero exit code:
Finally, start the server by running =ruby server.rb=: #+begin_src shell ruby server.rb #+end_src
Exporting this example starts the server, then waits for a second before quitting the process. Then, both the input block—which shows the command to start the server—and the output block—showing the server’s output—are exported to the HTML document:
ruby server.rb
Server running on http://localhost:4567...
To stop a stuck foreground process in Emacs (like the exporter that accidentally called a slow or continuous program), run C-g
to send an exit signal to the code that’s being executed.
For example, to show the output of the cat
command, use a shell
code block1:
#+begin_src shell :exports both :results output cat hello.txt #+end_src
Exporting this example to HTML produces a document with two code blocks in <pre>
tags. The first shows the command from the input code block (cat hello.txt
), and the second shows the results from running the command:
cat hello.txt
Hello, World!
Use the :prologue
and :epilogue
header arguments to prepare for code blocks to be run, without printing the setup and teardown commands to the exported file. For example, if the file to be read doesn’t exist, create it right before executing the code in the code block, and remove it after2:
#+begin_src shell cat new.txt #+end_src
Like before, this produces the both the input source block, and the results block with the contents of the file:
cat new.txt
Hello, new file!
Showing that a command fails by printing its error message is sometimes useful to illustrate a point.
Org’s exporter captures the command’s standard output stream to print to the output block, but it doesn’t capture standard error. For example, if the passed file does not exist, cat
returns an error. The exporter will not print the output block for the following example:
#+begin_src shell cat ghost.txt #+end_src
Since ghost.txt
does not exist, the exporter outputs a warning message while exporting and does not include the results block in the HTML result:
cat ghost.txt
To remedy this, make sure the output sent to standard error is redirected to standard output and that the code sample returns a zero exit code. The former is achieved by calling exec 2>&1
as the prologue to redirect the command’s output to standard output. To satisfy the latter, the command’s exit code is replaced with a zero by calling :
as the epilogue (which is a no-op that returns a zero exit code):
#+begin_src shell cat ghost.txt #+end_src
This example produces both the input and output code block in the HTML export:
cat ghost.txt
cat: ghost.txt: No such file or directory
The :exports
and :results
header arguments indicate the exporter should export both the code block and its results, and that it should print output of the code block as text results, respectively.
This example uses #+header
lines to split the header arguments over multiple lines, and is equivalent to placing all header arguments in the #+begin_src
line:
#+begin_src shell :exports both :results output :prologue "echo 'Hello, World!' >> hello.txt" :epilogue "rm hello.txt" cat hello.txt #+end_src
/usr/local/bin/aws s3 sync s3://jeffkreeftmeijer.com-log-cf logs
With the logs downloaded, generate a report by passing the logs to GoAccess to produce a 28-day HTML3 report:
find logs -name "*.gz" | \ xargs gzcat | \ grep --invert-match --file=exclude.txt | \ /usr/local/bin/goaccess \ --log-format CLOUDFRONT \ --date-format CLOUDFRONT \ --time-format CLOUDFRONT \ --ignore-crawlers \ --ignore-status=301 \ --ignore-status=302 \ --keep-last=28 \ --output index.html
This command consists of four commands piped together:
find logs -name "*.gz"
Produce a list of all files in the logs
directory. Because of the number of files in the logs directory, passing a directory glob to gunzip
directly would result in an “argument list too long” error because the list of filenames exceeds the ARG_MAX
configuration:
gunzip logs/*.gz
zsh: argument list too long: gunzip
xargs gzcat
xargs
takes the output from find
—which outputs a stream of filenames delimited by newlines—and calls the gzcat
utility for every line by appending it to the passed command. Essentially, this runs the gzcat
command for every file in the logs
directory.
gzcat
is an alias for gzip --decompress --stdout
, which decompresses gzipped files and prints the output to the standard output stream.
grep --invert-match --file=exclude.txt
grep
takes the input stream and filters out all log lines that match a line in the exclude file (exclude.txt
). The exclude file is a list of words that are ignored when producing the report4.goaccess --log-format CLOUDFRONT --date-format CLOUDFRONT --time-format CLOUDFRONT --ignore-crawlers --ignore-status=301 --ignore-status=302 --keep-last=28 --output index.html
goaccess
reads the decompressed logs to generate a report with the following options:
--log-format CLOUDFRONT --date-format CLOUDFRONT --time-format CLOUDFRONT
--ignore-crawlers --ignore-status=301 --ignore-status=302
--keep-last=28
--output=index.html
index.html
.
To sync the logs and generate a new report, run the sync.sh
and html.sh
scripts in a cron job every night at midnight:
echo '0 0 * * * ~/stats/sync.sh && ~/stats/html.sh' | crontab
On a mac, use Homebrew:
brew install awscli
Running the aws s3 sync
command on an empty local directory took me two hours and produced a 2.1 GB directory of .gz
files for roughly 3 years of logs. Updating the logs by running the same command takes about five minutes.
Since I’m only interested in the stats for the last 28 days, it would make sense to only download the last 28 days of logs to generate the report. However, AWS’s command line tool doesn’t support filters like that.
One thing that does work is using both the --exclude
and --include
options to include only the logs for the current month:
/usr/local/bin/aws s3 sync --exclude "*" --include "*2021-07-*" s3://jeffkreeftmeijer.com-log-cf ~/stats/logs
While this still loops over all files, it won’t download anything outside of the selected month.
The command accepts the --include=
option multiple times, so it’s possible to select multiple months like this. One could, theoretically, write a script that finds the current year and month, then downloads that stats matching that month and the month before it to produce a 28-day report.
GoAccess generates JSON and CSV files when passing a filename with a .json
or .csv
extension, respectively. To generate the 28-day report in CSV format:
find logs -name "*.gz" | \ xargs gzcat | \ grep --invert-match --file=exclude.txt | \ /usr/local/bin/goaccess \ --log-format CLOUDFRONT \ --date-format CLOUDFRONT \ --time-format CLOUDFRONT \ --ignore-crawlers \ --ignore-status=301 \ --ignore-status=302 \ --keep-last=28 \ --output stats.csv
My exclude.txt
currently consists of the HEAD
HTTP request type and the path to the feed file:
HEAD feed.xml
TAB
key to call org-cycle
, which cycles visibility for headers. Every TAB
press on a headline cycles through a different function1:
However, running Emacs with Evil mode in a terminal breaks the TAB
key for cycling through header visibility in Org mode.
Most terminals map both TAB
and C-i
to U+0009 (Character Tabulation)
for historical reasons, meaning they’re recognised as the same keypress. Because of this, there is no way to map different functions to them inside Emacs.
Evil remaps C-i
to evil-jump-forward
to emulate Vim’s jump lists feature2, which overwrites the default mapping for the TAB
key in Org mode.
To fix the tab key’s functionality in Org mode, sacrifice Evil’s C-i
backward jumping by turning it off in your configuration with the evil-want-C-i-jump
option.
This option needs to be set before Evil is loaded to take effect, so put it in the early init file:
;; Disable C-i to jump forward to restore TAB functionality in Org mode. (setq evil-want-C-i-jump nil)
With use-package, set the evil-want-C-i-jump
before the package is loaded by using the :init
keyword:
;; Install Evil and disable C-i to jump forward to restore TAB functionality in Org mode. (use-package evil :init (setq evil-want-C-i-jump nil) :config (evil-mode))
Vim and Evil don’t have a direct equivalent to fold cycling, but they have three different commands that achieve the same result:
zf
folds the headline’s subtreezo
opens the headline’s subtree to show its direct descendantszO
opens the complete subtree
Vim and Evil remember jumps between lines and files in the “jump list”. Because the jump locations are stored, you can use C-o
to jump to a previous location, and C-i
to jump back. For example:
}
in normal modeC-o
C-i
;; Elixir: elixir-mode (straight-use-package 'elixir-mode) ;; Language server client: Eglot (straight-use-package 'eglot)
Check out the repository for elixir-ls inside the ~/.emacs.d/
directory, and build1 it using mix elixir_ls.release
:
git clone git@github.com:elixir-lsp/elixir-ls.git ~/.emacs.d/elixir-ls
cd ~/.emacs.d/elixir-ls
mix deps.get
mix elixir_ls.release
In ~/.emacs.d/init.el
, add the path to the language_server.sh
file to the server programs list:
;; Add elixir-ls to Eglot's server programs list (add-to-list 'eglot-server-programs '(elixir-mode "~/.emacs.d/elixir-ls/release/language_server.sh"))
Evaluate the init.el
file by running M-x eval-buffer
(or by restarting Emacs) to reload the configuration.
Now, run M-x eglot
in any Elixir file to start the language server.
Because elixir-ls in a local repository, pull in changes through Git and rebuild the language server to update it in the future:
cd ~/.emacs.d/elixir-ls
git pull
mix deps.get
mix elixir_ls.release
<html> <head> <style> body{ font: 22px/1.6 system-ui, sans-serif; margin: auto; max-width: 35em; padding: 0 1em; } img, video{ max-width: 100%; height: auto; } </style> </head> <body> <h1>Hello world!</h1> </body> </html>
This output example purges the img
and video
selectors and minifies the stylesheet by removing newlines:
<html> <head> <style>body{font:22px/1.6 system-ui,sans-serif;margin:auto;max-width:35em;padding:0 1em}</style> </head> <body> <h1>Hello world!</h1> </body> </html>
The most popular tools to do this seem to be PurgeCSS for purging and cssnano for minifying, which are both PostCSS plugins.
We can’t use their command-line interfaces, as neither of these has a command-line option to take the internal stylesheet out of a page to purge and minify, as both expect the CSS and HTML to be in separate files. Instead, we’ll write a Node.js program that takes HTML files as input and prints them back out after minifying the internal stylesheets.
The program takes an HTML page through standard input with fs.readFileSync("/dev/stdin")
, which it then passes to a function named crush
:
#! /usr/bin/env node const fs = require("fs"); const crush = require(".."); let input = fs.readFileSync("/dev/stdin").toString(); crush.crush(input).then(console.log);
The crush
function takes the input HTML string and finds the stylesheet through a regular expression that captures anything within <style>
tags. For each stylesheet, it passes the match from the regular expression along with the whole input file to a function called process
. The promises it returns are collected and placed back into the <style>
tags in the HTML file:
const regex = /<style>(.*?)<\/style>/gs; exports.crush = async function (input) { let promises = [...input.matchAll(regex)].map((match) => { return process(match, input); }); let replacements = await Promise.all(promises); return input.replace(regex, () => replacements.shift()); };
The process
function handles processing the extracted stylesheet through PostCSS, initialized with the PurgeCSS1 and cssnano plugins. When the promise is fulfilled, the result—which is the purged and minified style sheet—is placed back into the <style>
tag:
function process(match, html) { return postcss([ purgecss({ content: [{ raw: html.replace(match[1], "") }] }), cssnano(), ]) .process(match[1]) .then((result) => { return match[0].replace(match[1], result.css); }); }
In the end, the index.js
file looks like this:
const postcss = require("postcss"); const cssnano = require("cssnano"); const purgecss = require("@fullhuman/postcss-purgecss"); const regex = /<style>(.*?)<\/style>/gs; exports.crush = async function (input) { let promises = [...input.matchAll(regex)].map((match) => { return process(match, input); }); let replacements = await Promise.all(promises); return input.replace(regex, () => replacements.shift()); }; function process(match, html) { return postcss([ purgecss({ content: [{ raw: html.replace(match[1], "") }] }), cssnano(), ]) .process(match[1]) .then((result) => { return match[0].replace(match[1], result.css); }); }
The program now takes an HTML document as a string through standard input and minifies the internal stylesheet3:
cat input.html | ./bin/crush.js
<html> <head> <style>body{font:22px/1.6 system-ui,sans-serif;margin:auto;max-width:35em;padding:0 1em}</style> </head> <body> <h1>Hello world!</h1> </body> </html>
When passing the stylesheet through PurgeCSS, the HTML page to check against is passed via the content
option:
purgecss({content: [{raw: html.replace(match[1], "")}]})
We need to make sure the stylesheet itself isn’t included in content
, as that would prevent PurgeCSS from removing the tags.
As an example, consider this input HTML file, which has styling for a <div>
which isn’t there:
<style>div { color: red }</style>
PurgeCSS should purge that CSS selector, because there are no <div>
tags on the page. However, if we pass the input file as a content
as-is, PurgeCSS will see “div” in the stylesheet and assume there’s a <div>
tag on the page.2
Upon closer inspection; PurgeCSS will also recognise the following document as having a <div>
tag for mentioning the word “div” in another tag:
<h1>An article about the div tag</h1>
--exec
option to run a shell command on every revision in the rebase. For example, to run a formatter like Prettier1 over each file in your repository for every past revision2:
git rebase -i --exec 'prettier --write {**/*,*}.js' ffcfe45
Being an interactive rebase, Git opens up your $EDITOR
to show the actions that are about to be executed. Although this rebase spans nine commits, there are eighteen actions as the commands are run as separate steps:
pick 22b042f npm install --save-dev mocha exec prettier --write {**/*}.js pick 76995a2 echo node_modules >> .gitignore exec prettier --write {**/*}.js pick 85d9d77 Use mocha as test script exec prettier --write {**/*}.js pick cfa165c Add failing test for crush.crush() exec prettier --write {**/*}.js pick 93cfd2a Minify with cssnano exec prettier --write {**/*}.js pick c9dcfb4 Add failing test for purging exec prettier --write {**/*}.js pick 1703abd npm install --save-dev @fullhuman/postcss-purgecss postcss exec prettier --write {**/*}.js pick 5faa328 Purge css with purgecss exec prettier --write {**/*}.js pick bc787ae Add bin/crush.js exec prettier --write {**/*}.js # Rebase ffcfe45..bc787ae onto 5faa328 (18 commands) # # Commands: # p, pick <commit> = use commit # r, reword <commit> = use commit, but edit the commit message # e, edit <commit> = use commit, but stop for amending # s, squash <commit> = use commit, but meld into previous commit # f, fixup <commit> = like "squash", but discard this commit's log message # x, exec <command> = run command (the rest of the line) using shell # b, break = stop here (continue rebase later with 'git rebase --continue') # d, drop <commit> = remove commit # l, label <label> = label current HEAD with a name # t, reset <label> = reset HEAD to a label # m, merge [-C <commit> | -c <commit>] <label> [# <oneline>] # . create a merge commit using the original merge commit's # . message (or the oneline, if no original merge commit was # . specified). Use -c <commit> to reword the commit message. # # These lines can be re-ordered; they are executed from top to bottom. # # If you remove a line here THAT COMMIT WILL BE LOST. # # However, if you remove everything, the rebase will be aborted. # # Note that empty commits are commented out
When running the rebase, Git halts whenever the command returns a non-zero exit code to allow you to check what happened. In this example, the first couple of commits don’t have any JavaScript files to format, which are skipped with git rebase --continue
:
Executing: prettier --write {**/*,*}.js [error] No files matching the pattern were found: "**/*.js". [error] No files matching the pattern were found: "*.js". warning: execution failed: prettier --write {**/*,*}.js You can fix the problem, and then run git rebase --continue
Each commit gets picked before the execution happens, which can cause conflicts between formatted and still-unformatted code. In that case, remember that the next exec
step reformats the code and amends it to the commit, so you can pick the unformatted version and have the formatter do the formatting.
: Examples with different formatters:
mix format
(Elixir)git rebase -i --exec 'mix format' main
git rebase -i --exec 'rubocop --auto-correct' main
git rebase -i --exec 'prettier --write {**/*,*}.js' main
: ffcfe45
is the commit that adds prettier to the project. I injected that commit after the others were already written to be able to the formatter retroactively to clean up history.
git init touch "file.txt" git add file.txt git commit -m "Subject" -m "First paragraph" -m "Second paragraph"
git log --format=medium
The “medium” format is the default when passing no --format
option:
git log
commit f142f6d3327fbc17ff3f9a7f6c4157d70da7083f Author: Alice <alice@example.com> Date: Thu Jul 22 07:45:25 2021 +0200 Subject First paragraph Second paragraph
git log --format=oneline
git log --format=oneline
f142f6d3327fbc17ff3f9a7f6c4157d70da7083f Subject
git log --format=short
git log --format=short
commit f142f6d3327fbc17ff3f9a7f6c4157d70da7083f Author: Alice <alice@example.com> Subject
git log --format=full
git log --format=full
commit f142f6d3327fbc17ff3f9a7f6c4157d70da7083f Author: Alice <alice@example.com> Commit: Bob <bob@example.com> Subject First paragraph Second paragraph
git log --format=fuller
git log --format=fuller
commit f142f6d3327fbc17ff3f9a7f6c4157d70da7083f Author: Alice <alice@example.com> AuthorDate: Thu Jul 22 07:45:25 2021 +0200 Commit: Bob <bob@example.com> CommitDate: Thu Jul 22 07:45:25 2021 +0200 Subject First paragraph Second paragraph
git log --format=email
git log --format=email
From f142f6d3327fbc17ff3f9a7f6c4157d70da7083f Mon Sep 17 00:00:00 2001 From: Alice <alice@example.com> Date: Thu, 22 Jul 2021 07:45:25 +0200 Subject: [PATCH] Subject First paragraph Second paragraph
git log --format=raw
git log --format=raw
commit f142f6d3327fbc17ff3f9a7f6c4157d70da7083f tree bdd68b0120ca91384c1606468b4ca81b8f67c728 author Alice <alice@example.com> 1626932725 +0200 committer Bob <bob@example.com> 1626932725 +0200 Subject First paragraph Second paragraph]]>
--message
or -m
option multiple times:
git commit -m "Subject" -m "First paragraph" -m "Second paragraph"
The first message is the subject line, and every subsequent one becomes a paragraph:
git log
commit 3c2a65bd0e77bc323b77463ced0feb7c02f3acac Author: Alice <alice@example.com> Date: Sat Jul 17 13:05:58 2021 +0200 Subject First paragraph Second paragraph]]>
git-filter-repo
library supports many ways of file filtering and history rewriting1, but extracting directories and files are two I’ve needed in the past.
To extract the directory named path/to/dir
, move all files in path/to/dir
to the repository root:
git filter-repo --subdirectory-filter path/to/dir
To extract a single file named path/to/file.txt
, move all files in path/to
to the repository root:
git filter-repo --subdirectory-filter path/to
Then, remove all files except file.txt
:
git filter-repo --path file.txt
Git comes with a file named git-prompt.sh
, which exposes __git_ps1
—a function that returns the current Git branch name—to be used in a terminal command prompt2.
The instructions in the file encourage copying the file somewhere like ~/
, and then using it in your prompt. However, I prefer being able to pull in changes by keeping it in a Git repository. Keeping a local checkout of Git’s source proved a bit too bulky, so I decided to extract the file I needed into a separate repository3.
Extracting a single file from a repository requires extracting a directory and then removing any unwanted files. First, we clone Git’s repository:
git clone https://github.com/git/git.git
Cloning into 'git'... [...] Updating files: 100% (3956/3956), done.
Then, switch directories to end up inside:
cd git
git-filter-branch
An often-recommended approach is using git filter-branch
. To extract the contrib/completion
directory, we pass the directory name through the --subdirectory
option:
git filter-branch --subdirectory-filter contrib/completion
The command halts, refuses to continue and displays us the following warning:
WARNING: git-filter-branch has a glut of gotchas generating mangled history rewrites. Hit Ctrl-C before proceeding to abort, then use an alternative filtering tool such as 'git filter-repo' (https://github.com/newren/git-filter-repo/) instead. See the filter-branch manual page for more details; to squelch this warning, set FILTER_BRANCH_SQUELCH_WARNING=1.
Aside from the risk of corrupting your repository, running git-filter-branch
on this repository takes well over a minute if we were to run it with the FILTER_BRANCH_SQUELCH_WARNING
environment variable set4.
It’s generally a bad idea to use git-filter-branch
, so we’ll have to find another way.
git-filter-repo
As instructed, we’ll use git-filter-repo
instead. The git-filter-repo
subcommand isn’t bundled with Git, but it’s installable through most package managers5. On a Mac, use Homebrew:
brew install git-filter-repo
==> Downloading https://ghcr.io/v2/homebrew/core/git-filter-repo/manifests/2.32.0-1 [...] /usr/local/Cellar/git-filter-repo/2.32.0: 8 files, 278.2KB
The command to extract a subdirectory is a direct translation from the one in git-filter-repo
. Again, we pass the directory we’d like to extract as the --subdirectory-filter
:
git filter-repo --subdirectory-filter contrib/completion
Parsed 65913 commits HEAD is now at 5a63a42a9e Merge branch 'fw/complete-cmd-idx-fix' New history written in 13.23 seconds; now repacking/cleaning... Repacking your repo and cleaning out old unneeded objects Completely finished after 15.47 seconds.
Voilà! That was at least five times as fast. We’ve extracted all files in the contrib/completion
directory, while retaining their history:
ls
git-completion.bash git-completion.tcsh git-completion.zsh git-prompt.sh
The previous example removed most of git’s source code from our checkout, but we’re still left with the completion files in the directory. In this specific cace, we only really need contrib/completion/git-prompt.sh
.
To extract a single file from a git repository, first extract the subdirectory like we’ve just done, then use the --path
option to filter out all files except the selected one:
git filter-repo --path 'git-prompt.sh'
Parsed 1240 commits HEAD is now at 9db4940 git-prompt: work under set -u New history written in 1.86 seconds; now repacking/cleaning... Repacking your repo and cleaning out old unneeded objects Completely finished after 2.62 seconds.
Now we’re left with a repository holding a single file.
ls
git-prompt.sh
You can pass the --path
option multiple times to take out more than one, use --path-blog
or --path-regex
to match multiple files with a pattern, or combine any of these with the --invert-paths
option to invert the selection to remove the matching files instead of keeping them.
Outside of extracting subdirectories and files, git-filter-branch
can move files between directories, remove files from repositories, move a while repository into a subdirectory, rewrite commit messages, change author names, and more.
I extracted the git-prompt.sh
file out of Git’s repository back in 2016 as git-prompt.sh, and I’ve been using this prompt ever since:
source ~/.config/git-prompt.sh/git-prompt.sh export PROMPT='%~ $(__git_ps1 "(%s) ")$ '
While in a Git repository, my prompt shows the current branch name:
$ ~/.config/git-prompt.sh (main)
Assuming I’m the only one using this extraction, I must admit there’s a problem with this approach. While it seems like it would save some time because it’s quick to pull a new version of the file, updating it involves extracting the file, then rebasing the license and readme onto Git’s upstream main branch before I get the ease of pulling in changes through Git. If no such extraction existed (I’m comitted to it now), it would have been quicker to download the file directly through cURL:
curl https://raw.githubusercontent.com/git/git/master/contrib/completion/git-prompt.sh --output ~/.config/git-prompt
A mirror that automatically updates and extracts the file would solve this. Until then, it’s a nice example to show how to extract files from Git repositories.
The documentation page for git-filter-branch
reiterates that using this command is “glacially slow”, that it “easily corrupts repos”, and urges us once more to use git-filter-repo.
Check out the documentation for installation instructions.