Bowmansarrow

Motivation

For work, I often find myself writing documents for the projects I'm on. Usually, I just open Emacs and start a LaTeX file, but this time I thought I'd try using Org mode for writing the document. There were a couple of motivations for this, in one case, I needed to generate a presentation and I'd seen enough posts about exporting to a PDF via Beamer to finally take the plunge and try it for myself. In the second case, I had been taking a bunch of notes and computing metrics in tables.

Beamer

Getting started with this document took a little longer because I didn't know what I was looking for. I needed to add #+STARTUP: Beamer to the top of my file and then press C-c C-c on that line to get the Beamer export options. The other confusing thing about exporting a presentation with Beamer is how the headlines work. Most of the examples I saw used #+OPTIONS: H:2 or #+OPTIONS: H:3. This sets the header level to export.

#+options: H:2 * header 1 This header is not exported ** header 2 This header is exported as the title of a frame *** header 3 This header is exported as a subsection on a frame

Once I figured out how that structure worked, I made quick progress on my presentation.

The themes available look nice, however, for a lot of them, I couldn't tell a difference between one and the next. I looked at the Beamer theme matrix, each row represents a theme, while each column represents a different color scheme. Looking across one row, several of the color schemes look identical to me. I hope I'm not missing something fundamental, but there doesn't seem to be a lot of variation. What is provided does look good. I chose to use the Dresden theme with the lily color scheme.

PDF Documents

This one was more straight forward. Out of the box, the result is pretty good. There are some tweaks to be made. Adding things to the preamble is as simple as adding another #+latex_header: option. In my case, I added the following:

#+latexheader: \usepackage{array} #+latexheader: \setlength{\parskip}{1em} #+latex_header: \newcommand{\tabcap}[2]{\caption{\label{tab:#1}{\textsc{\scriptsize #2}}}}

I have several tables in this particular document and I wanted to change the default font and size for the caption, so I provided a new command for that purpose. The array package is also used for the tables to help with column layout. I want a blank line between paragraphs, so I change the default parskip value.

Most of the document is based around the data I present in the metrics tables, so I wanted to control a couple aspects of the layout. Each table has a #+NAME:, a #+CAPTION: and some layout options specified with #+ATTR_LATEX:. The only challenge I did not overcome was the placement of the table caption. On export, the caption appears above the table, but I would prefer it to be below the table. I did not figure out how to adjust the placement of the caption, so I manually modified the tex. The downside of that approach is now I have to remember all the manual changes I make if I choose to maintain the org file and export again. In this case, I just moved to modifying the tex file instead of the org file.

Conclusion

I frequently use Org mode for managing my day at work. I track what tasks I'm working on, meetings to attend, time spent in each, and general note taking for each. It is nice to be able to section parts of the file into logical sections and be able to quicly navigate between them. Using the same power when writing documents to distribute as PDF files or as presentations is just as rewarding. You can even setup the org file for different kinds of exports by using C-c C-e # and choosing a template of options to insert. In the case of a Beamer file, remember to add the appropriate startup option and evaluate it first so the template exists to insert. In general, Org mode is a great place to start writing documents, or even finish them if you don't need to modify htem manually in the external format.

Tags: #emacs #org #pdf #beamer

Motivation

Recently, I looked into other package management tools for Emacs. I looked at ElPaso, Borg, and Straight. They all seem to be ways to avoid using the Emacs built-in package.el, which I'm not all that interested in doing. I find the simplicity of being able to list the available packages I might be interested in installing too useful to switch away to something else. They are interesting projects, and you should probably look into one or more of them to see if they make sense for your use.

Even though I was already using use-package, and am generally happy with how it works, I thought it might be fun to re-organize and rewrite my configuration. There is a lot of cruft and I needed to sort that out anyway.

For goals, I chose this list:

  • Rewrite without use-package.
  • Keep start-up time to a minimum (my startup time was just above 2 seconds).
  • Programmatic activation of chosen modules with byte compilation.
  • Prefer versioned packages over “latest commit” packages (ie prefer other repositories versions over MELPA versions).

Influence

As mentioned previously, I used Prelude for a while and still take inspiration from its code and layout.

As a summary, Prelude is organized around some core configuration with some modules which can be configured or mixed in automatically depending on the mode needed for editing a file. For example, editing a clojure file would automatically activate the Clojure module in Prelude. This is a cool feature, but I'm not that ambitious, so it is not one I tried to emulate in my configuration. However, how the core and modules are organized, I did take a bit of inspiration from that.

Approach

Following Prelude's lead, I have the following directories:

  • core: This is where the base configuration exists. It contains the elisp needed to install, activate and configure the base set of packages I always want to be enabled.
  • modules: This is the list of “modules” for various programming modes or other tools which I want to use, but not necessarily have activated on startup. For example, I might be interested in editing the Lua configuration for my AwesomeWM configuration, but since I do that rarely, I don't want it activated unless I edit a Lua file. This directory is not on the load-path. More information below.
  • personal: This is where personal configuration goes. Most of the configuration will reside in either the mycustom.el file which is for general customization of various modules or for things that just don't fit anywhere else. The mymodules.el file is where I activate the modules I want to use. This directory is on the load-path, modules are symlinked here (or when I am on a Microsoft Windows machine, copied here).
  • sample: This folder contains an example mymodules file to copy into the personal directory and edit there.

I make aggressive use of autoload and with-eval-after-load to control how and when packages are enabled. There is still more to do, I think using mode hooks would be a cleaner approach for some things, so I'm still considering how I might implement that concept, especially for programming modes and LSP.

Packages

This is how I configure my package-archives, I have it listed in the order I prefer to get packages, but the ordering here is not relevant.

(setq package-archives
   (quote
    (("gnu"          . "https://elpa.gnu.org/packages/")
     ("nongnu"       . "https://elpa.nongnu.org/nongnu/")
     ("melpa-stable" . "https://stable.melpa.org/packages/")
     ("melpa"        . "https://melpa.org/packages/")
     ;; [2021-03-31 Wed] org currently at 9.4.5, but this is about to
     ;; go away after the release of org 9.5. Org will be distributed
     ;; through GNU elpa and org-contrib will be distributed through
     ;; nongnu elpa (see above for both).
     ("org"    . "https://orgmode.org/elpa/"))))

I continue to use the orgmode.org/elpa until the release of Org 9.5, and will remove it from the list at that time.

To use melpa and still prefer versions, I use a feature of package.el called priorities. If a package exists in a higher priority elpa, then install from there, otherwise, try the next lower priority elpa and so on until the package is found. Some packages are only found in melpa so I get them from there. This allows me to get a versioned package from one of four other elpas before trying melpa

N.B. elpa in the last paragraph refers to the Emacs Lisp Package Archive specification rather than a specific repository.

This is how I configure the priorities for packages, higher numbers are preferred over lower numbers when retrieving packages:

(setq package-archive-priorities
      '(("gnu"          . 99)
        ("nongnu"       . 80)
        ("org"          . 75)           ; see comment above, this will
                                        ; be going away
        ("melpa-stable" . 70)
        ("melpa"        . 0)))

I don't want all of the packages I have installed activated by default, so I pass t to package-initialize to not activate anything and allow the rest of my configuration to do that work.

(if (file-exists-p package-user-dir)
    ;; don't activate all packages, instead they will be added via
    ;; mypackage-require, which will install if needed as well.
    (package-initialize t)
  ;; on a new system, need to activate everything initially
  (package-initialize)
  (package-refresh-contents))

Finally, Prelude has a nifty idea of tracking a default set of packages that should be installed initially, and to which additional packages are added as they are activated later. I cloned that idea directly from Prelude and use it as well. I do attribute the code and give credit to the author in my code. I did change some of the names to match the rest of my code (ie mypackage- instead of prelude-).

Modules

The modules portion of my config contains specific configuration files for each of the programming, text, and utility (ala docker) modes I currently use. The mymodules-config.el has functions that create symlinks in the personal directory and then byte compiles the file there. There are two variables used which can be configured using setq:

  • mymodule-available-dir: This is the directory where the modules live. They are symlinked or copied from here.
  • mymodule-active-dir: This is the directory where the modules are symlinked (or copied) to make them active.

To create the symlinks, I thought a macro would make sense:

(defmacro mymodule-activate (module)
  "Install symlink to source and compiled MODULE files

Gets MODULE from MODULE-AVAILABLE-DIR to create a symlink in
MODULE-ACTIVE-DIR, byte compiles the module after creating the
symlink and then loads the MODULE. If the MODULE is already
installed, just load it."

  ;; initially create names for use in the macro itself. This is not
  ;; really needed as the only input is MODULE which is a string, but
  ;; it means the symbols created can *only* be used here in this
  ;; macro.
  (let ((module-name (make-symbol (file-name-base module)))
        (load-name (make-symbol "load-name"))
        (available-module-file (make-symbol "available-module-file"))
        (active-module-file (make-symbol "active-module-file")))
    `(let* ((,module-name ,module)
            (,load-name (file-name-base ,module))
            (,available-module-file (expand-file-name ,module-name
                                                      mymodule-available-dir))
            (,active-module-file (expand-file-name ,module-name
                                                   mymodule-active-dir)))
       (if (file-exists-p ,active-module-file)
              (load ,load-name t)
            (if (string= "windows-nt" system-type)
                ;; if this is Microsoft Windows, we can't use symlinks, so
                ;; copy the file instead.
                (copy-file ,available-module-file mymodule-active-dir)
              ;; Linux or Apple Mac can handle symlinks
              (make-symbolic-link ,available-module-file
                                  mymodule-active-dir
                                  t))
            (byte-compile-file ,active-module-file)
            (load ,load-name t)))))

However, since symlinks don't work in Microsoft Windows, I have to check for that and just copy the files from the available directory to the active directory.

Customization

Out of the box, I don't need to do anything. If I don't have a mymodules.el file, then just the default packages are loaded and I can still get a lot of work done. Things like Org-mode, Magit, Ivy, etc. allow me to do enough from day-to-day. I tend to spend a lot of time in Org-mode tracking work, tasks, meetings, notes, etc. so the base is sufficient.

However, as I also need to edit Java, Json, XML, and other things like Docker files and diagrams, so I activate those modules on the machine where I do that work. This gives me the flexibility to install the modules I use to configure Emacs for the machine I'm on at the time. My work configuration contains different modules than the configuration on the client environment, and from my personal computer.

Conclusion

This has been an interesting exercise! It took about eight hours to accomplish the rewrite to the point I have it now. I did clone some code from Prelude, and I did reach out to Bozhidar for permission, which he graciously provided.

There are still some things I think could be improved in this configuration. For example, I need to look into using mode hooks more frequently. Some of the autoload code does not activate the way I think it should, but a mode hook would most likely fix that problem.

As for meeting the goals:

  • I have replaced use-package. To do so, I used pp-macroexpand-expression to understand what was happening and then implemented a solution using either or both of autoload and with-eval-after-load.
  • Startup time is about 1.6 seconds.
  • The macro module-activate handles making symlinks or copies, byte compiling the module and loading it.
  • Using package priorities, I get versioned packages, or a version from MELPA if that is the only repository where a package is distributed.

The code: git repo on bitbucket.

Tags: #emacs

I was watching this video on the System Crafter's YouTube channel during which he considers whether or not using a pre-made Emacs config is a good thing. He has several good points on both the pros and cons sides of things. I won't mention all of them, you should go watch the video for that, but here are a few things that resonated with me.

Pros

  • Makes getting started easier. I know from experience, configuring Emacs to do cool stuff can be a bit of a daunting challenge when first starting. I've been using Emacs for a number of years, hand-coding my configuration and trying to get it to do the cool things I saw others doing on various videos. Now it is something I enjoy a bit more, but I eventually ditched my hand-coded configuration and went with a pre-built config for a while (more on this in a moment). Things worked out of the box in a nice clean way. So, even though I “started” kind of in the middle of my experience, this reasoning makes sense to me.

  • Better integration between different packages. If there is one thing I have struggled with and have seen in posts on reddit and stackoverflow, its the many ways to do really common things, like completion (ac or company?), fuzzy stuff (ivy, helm, consult, selectrum, counsel…??), language tools for software development (slime, sly, lsp… ???). A lot of the pre-built configurations just make decisions on these and then make them work.

  • Often there are more learning resources. Yes. In the cases I've reviewed (and by reviewed, I mean the ones whose documentation I read when I was deciding which one might be most to my way of thinking and working), there has been fairly good to amazingly good documentation. The approach is usually how to work with the configuration to make the smallest changes needed to make it “yours” and leave the rest alone. Each claims to provide sane defaults, which I find largely to be true for some definition of sane.

Cons

  • It might convince you that vanilla Emacs is too hard to use. The idea here lines up with the next one. Basically, if you look at the code base for most of the pre-made configurations, they are not small. So, it may lead you to believe this is the minimal viable source needed to actually configure Emacs to be usable. This is not true. A good friend of mine has been using Emacs for a number of years and his configuration is very short. He doesn't configure a lot of the whiz-bang features so popular in other IDEs, editors or even within Emacs itself. Works fine for him. Emacs, without any configuration or additional packages is still a very powerful tool. It just depends on what you are used to and what your expectations are. The more “things” you expect Emacs to do, the more you need to do to configure Emacs to work that way.

  • Specialized configurations can make things more confusing. One of the arguments both for and against using a pre-made configuration is that it gives you a way to see how someone else put things together so you can learn from it and then, eventually, write your own. But trying to read the code for some of the pre-made configurations can be just like those tween choose-your-own adventure books from the 1980's. Trying to understand why something works in that configuration but not so much when you try to write it in your own custom configuration can be confusing. It definitely requires you to learn a lot, but can lead to enough frustration to rage-quit from time to time. Additionally, you need things like a pre-made configuration provides to easily navigate through a call stack in source code when you are reading it.

My journey

I started using Emacs a long time ago, I was fascinated by the Lisp family of languages and thought using Emacs would help me learn and use that particular language. While that is absolutely true, Emacs Lisp is just another dialect and knowing it is not the same as knowing Common Lisp or Scheme. But I digress.

I was also very interested in how productive people were when using Emacs and I wanted to that for myself as well. I did learn enough Emacs Lisp to sort-of make things kinda work and I was generally happy with most of my work flow being in Emacs and jumping out to other tools for more hard-core editing. I just kept coming back to Emacs because I found myself more comfortable here.

I started with Prelude

I really wanted some of the features I'd seen in other IDEs and editors, and so I researched a few of the pre-built configurations and settled on prelude by the venerable Bozhidar Batsov. It should be noted, at the time I was looking, there weren't that many pre-built configurations. Spacemacs was around, I think Doom Emacs was just getting traction, the Emacs Starter Kit existed… those are the ones I recall reviewing. I didn't do an exhaustive search, so there might have been others. I chose prelude because I wanted something as close to vanilla Emacs as possible and something whose code I could use as an example to learn from. I didn't want Vi in Emacs, I can use Vi if I really want that experience, so nothing against Vi, just not what I wanted to use. I didn't want a bunch of hijacking macros around things like package.el. And I didn't want a literate configuration. Org mode is super cool, and I definitely find it to be one of the most useful packages I use, but I wanted my configuration to be straight code. I can write comments in my code just as well as writing text around code snippets, and with hide-show-mode I can collapse things if I want to. I wanted to be able to use the existing features of Emacs and maybe a few well integrated packages. I got that from prelude.

And then I wrote my own

After using it for a while, and learning from how it was written, I wrote my own configuration. I setup how my configuration was organized in much the same way that prelude did it, although without quite the same level of sophistication. My only complaint was the 19 seconds it took to startup. That was something I just thought I'd have to live with in the post-prelude world.

And then I did it again

I started seeing a lot of people moving their configuration to use-package most often citing the simplicity of configuration and the speed at which Emacs was starting. So, I took the plunge and rewrote my entire configuration using use-package to install, load and configure the packages I was using. For fun, I went back to using prelude, reconfigured a few things, and got my start time to around 4 seconds which compares favorably with my current configuration that starts in just over 2 seconds. Slow by some people's standard, but I start it in daemon mode and the use emacsclient so I don't keep starting/restarting Emacs, and starting emacsclient is nearly instantaneous. Still, I think there are a few things I could tweak a bit to get the prelude version to start faster and shave off a couple (or more) of those seconds.

So those pre-made configurations have value

From my experience, I learned a lot from starting over with my configuration by using prelude and then growing back into a personal configuration heavily inspired by how prelude is written. I have also looked at the source for Spacemacs and Doom Emacs and even Emacs Starter Kit and there is a lot to learn from those as well. I think the “pros” they provide out weigh the “cons” enough to recommend them. I would suggest identifying what you would want from Emacs, then reviewing the goals and source code for several pre-built configurations before just jumping on one and being frustrated that it doesn't do feature x in some way you think it should. Try looking on the Emacs Wiki for a list of pre-made configurations that might make sense to you.

And Bozhidar, if you happen to come across this page, thank you very much for your work on Prelude. I've learned a lot.

Tags: #emacs

Time to start over… again…

A while back, I posted I was unhappy with the editing facilities of the blogging site I was using at the time. I worked with it anyway trying a few different solutions including using Jekyll, using Org-Mode's export to HTML and tinkering with that, just typing using the built-in HTML editor, and finally trying to use elisp to use the Blogger API. That was a fantastic disaster which I won't go into here, lets just say after about four hours of just trying to get authenticated using the oauth2 elisp library and Google's ideas of how that should work, I rage quit and went to read a book.

It was just too painful to get anything done with that site, so I posted a question on reddit and received several very good answers. I am truly grateful for the Emacs community on reddit. My experience there has always been positive with very helpful people. One comment suggested using writefreely.el as the simplest solution. That comment also suggested Weblorg which I tried and generally liked, but after a couple of hours I realized I still had a long way to go before I would have a working solution, and I went back to try WriteFreely.

Okay, that was dead simple to use. The writefreely package in Emacs just works, the result looks good and it took almost two minutes of my time to setup and another 30 minutes of tinkering with a few things to make sure I understood them and tweak my configuration (which is nothing more than a snippet to start my org file). So, props to pxoq for the great suggestions. Also thanks to the other comments left by other users, I did mindfully read and research all of them.

If you are looking for some approaches to blogging, writing posts in Emacs and publishing them somewhere on the web, read through the comments on that reddit post, there are some truly great and simple approaches offered. Much thanks to all of them for responding.

Tags: #writefreely #emacs

The problem…

A friend of mine sent a request on Slack today, he asked if anyone had tried to solve a particular problem he had found posted on twitter. The post references this GitHub page where the problem and some of the solutions are described. My friend specifically asked if anyone had solved this with pure DataWeave, a transformation language embedded in the MuleSoft software known as Mule.

I haven't coded anything in DataWeave for quite a while, so I thought I'd give it a go. I had to read up on some of the functions, and then use this REPL to do the work. I copied the example input from the GitHub page, and started hacking away.

A first pass…

At first I thought I could use some nifty recursion and a “stream” data structure. Basically, that is a pair, the first part is the current value, the second is a function which returns a pair with the next value and a function. That didn't work out because, well, recursion. DataWeave is not tail-call optimized and will overflow the stack at 256 frames.

Bummer.

A second pass…

Okay, so looking at it from a different perspective, I just needed to setup the indices correctly and then I could iterate over the payload and come up with a solution.

%dw 2.0
output application/json
var allWithReset = zip(payload map $.reset_lesson_position,  payload.lessons map $$ )
var allStartVals = allWithReset map if ($[0] == true) 0 else $[1] + $[1]
---
payload map (value, index) -> {
    title: value.title,
    reset_lesson_position: value.reset_lesson_position,
    position: index + 1,
    lessons: value.lessons map {
        name: $.name,
        pos: allStartVals[index] + ($$ + 1)
    }
}

That did the trick… with the 3 records in the example data set. But I decided to add another record. And, well, you can guess what happened next… yep, the indices were off – completely. Obviously, the problem is how I was creating the indices, so let's look there for a solution:

%dw 2.0
output application/json
var allWithReset = zip(payload map $.reset_lesson_position,  payload.lessons map $$ )
var allStartVals = allWithReset map if ($[0] == true) 0 else $[1] + $[1]
var allStartNormalized = allStartVals map if (allStartVals[$$ - 1] == 0) 2 else allStartVals[$$]
---
payload map (value, index) -> {
    title: value.title,
    reset_lesson_position: value.reset_lesson_position,
    position: index + 1,
    lessons: value.lessons map {
        name: $.name,
        pos: allStartNormalized[index] + ($$ + 1)
    }
}

Yep, that fixed it! Well, at least until I added another record and changed one of the reset values… And then the indices are broken again. So, the solution is just to add more complexity, right!??!

%dw 2.0
output application/json
var allWithReset = zip(payload map $.reset_lesson_position,  payload.lessons map $$ )
var allStartVals = allWithReset map if ($[0] == true) 0 else $[1] + $[1]
var allStartNormalized = allStartVals map 
    if (allStartVals[$$ - 1] == 0 and allStartVals[$$] == 0) 
        0 
    else if (allStartVals[$$ - 1] == 0 and allStartVals[$$] != 0) 
        2 
    else allStartVals[$$]
---
payload map (value, index) -> {
    title: value.title,
    reset_lesson_position: value.reset_lesson_position,
    position: index + 1,
    lessons: value.lessons map {
        name: $.name,
        pos: allStartNormalized[index] + ($$ + 1)
    }
}

Nope, that fails too. At this point I start getting a little banter in the slack channel. We discuss the merits of different approaches to a solution, then one of my friends posts a working solution that is elegant and, after thinking about it, somewhat obvious. As a matter of fact, after saying recursion several times, I realized I needed to be reaching for the reduce function, which is how DataWeave manages recursion.

The solution…

Nope, not going to post it here, you'll have to go read this blog post to get the answer. This guy writes some nice blog posts as well, so feel free to peruse and learn a few things. Good stuff!!

Tags: #MuleSoft #Dataweave