documenting small tools, creating new markup languages, and using the shell
I am Alvin and this is a blog that collects various posts that were written as I was learning to program and use the shell.
More generally, I am a writer with over twenty years experience crafting copy for technical documentation, technical blogs, corporate trainings, mobile apps, product marketing, email campaigns, business strategy, and arts / entertainment.
Have you ever navigated to a directory in your terminal and wanted to select a file or folder from a menu, rather than typing ls to print a list of contents? Or perhaps you would rather use a single command to present a list of git options for you to choose instead of having to search your command history for frequently used (but often forgotten) commands? If you have experienced these annoyances, or if you want a simple way to choose items from a menu, fzf is the tool for you.
In this article you will learn about fzf, the command-line fuzzy finder. You will also learn a few ways that fzf can improve your scripts.
fzffzf can be installed by cloning the repository, using homebrew on MacOS, or by using the default package manager in your Linux system. For more information, see junegunn/fzf#installation.
fzf can be used to generate a simple menu from a list of items. For example, you can use fzf to display a list of files for you to choose from. To do this you can pipe a list of items into fzf in your terminal by using the command below
find . -type f -maxdepth 1 | fzf
You can also use command substitution to open the selected file in your editor.
$EDITOR $(find . -type f -maxdepth 1 | fzf)
Now that you understand the basic use of fzf, you can use the tool for more interesting automations.
fzf with GitLet's say you have a list of git commands that you use regularly but you want to speed up your git usage by choosing commands from a menu, rather than typing git –help to remember the name of the command you want to use. First, start by printing a list of your commands in your terminal.
# The "-e" option translates "\n" to a literal new line, required by fzf echo -e "clone\ninit\nadd\ncommit\npush\n"
Once your list of commands is set, you can pipe this list to fzf to display them in a menu
echo -en "clone\ninit\nadd –all\ncommit\npush\n" | fzf
The command above will present your list of commands as a menu, allowing you to select a command via fzf. However, this does not actually run the git command. To use fzf for executing git commands, you can use the code below
git $(echo -en "clone\ninit\nadd –all\ncommit\npush\n" | fzf)
The above command will run the selected git command without any additional work on your part, simplifying your workflow and saving time.
In this article you learned about fzf, how to install fzf, and some basic usage examples you can apply to your workflow today. For more information about the fzf command-line fuzzy finder, visit the official fzf repository at https://github.com/junegunn/fzf.
Name: ??
Goal: Create a somewhat restricted markup language with minimal formatting noise.
Lang: ???
Many of the ideas below were borrowed from other systems / markup languages.
% Comments run to the end of the line and are ignored during processing
= h1 == h2 === h3
---
+ item 1 + item 2 ++ item 2.1 ++ item 2.2 +++ item 2.2.1
# item 1 # item 2 ## item 2.1 ## item 2.2 ### item 2.2.1
``const addTwo = (num) => num + 2``
`` const addTwo = (num) => num + 2 addTwo(5) // 7 ``
*bold* |italic| _underline_ ~strikeout~
> this is text in a block quote > each line in the block quote must be > prepended with the right bracket char (like markdown) > otherwise the text is assumed to be paragraph
::title=https://url.com
::=https://url.com
!!=image.jpg
!!=image.jpg::https://viewimage.com
:highlight: ^superscript^ ,subscript,
markup |= special formatting used when creating textual documentation
SSIMS := Silver Spring International Middle School
A footnote|1 can be used like so --- 1: Here is some supplemental text about footnotes
=Cities, Pop, State= +Balitmore, 600K, Maryland+ +New York, 2.5M, New York+
--- title: A Markup Language author: Unforswearing ---
--- title: Appendix include: pub_data.yaml --- % this document now contains all front matter located in pub_data.yaml
= This is a document that contains HTML tags Below is an introduction using a specified <div> element #div Hello, I am unforswearing and this text is in a div element #p Here is an additional paragraph
see also: configuration language, command language, other non-turing complete langauges
a less minimal list could be something like:system commands: args, print/show, input/read, exit, execute, (others) math: basic infix expressions / equality / modulous / etc <<< implemented myself ?? control: if/then/else, for..in, break, return, try/err, wait/sleep, loop data structures: array/list (and methods) , struct/object {} (and methods), function/func/fun/fn/int/ types: number, string, array, struct, function, etc
use("/path/to/lib_file.lang")
..file lib.lang
fn printhello() show "hello" dn
exp(printhello)
..file script.lang
use("lib.lang")
@printhello()
From Gleam
It would be fun to implement some of the Open Formula spec
list.char - split text or file by character, eg. `{"a",,"l","i","s","t"}
list.word - split text or file by word , eg. {"a", "list"}
list.line - split text or file by line , eg. {"a list"}
file - method to save files to disk , eg. @conv({#text, file, "/path/file.txt"}
these operations only require two arguments:
@conv({#var, file, "/path/to/number.txt"})
@conv({#var, file, "/path/to/text.txt"})
examples
..
..change type from number/file/list/text to text/list/file/number
..NOTE: conv will only accept a list as an argument!!
@conv({#var, type - list|number|text|file, #optional_argument})
..
..NOTE:
..- there is no number or none types so they cannot be declared as classes like list, text, and file
..- file to text is not necessary since files are read by default
.. when declaring the file instance using `filename::file("/path/to/file.txt")`
..
.."file to list" and "text to list" splits the file or text by character, word, or lines
..text and file have the same properties so the operations will work exactly the same for both types
@conv({#var, list, list.char})
@conv({#var, list, list.word})
@conv({#var, list, list.line})
..
.."number to file" and "text to file" saves the var value into a new file
.. at path specified in #optional_argument
@conv({#var, file, "/path/to/number.txt"})
@conv({#var, file, "/path/to/text.txt"})
..
..number to text conversions numeric characters to text strings
..the third item in the list argument will be discarded
@conv({#var, text})
..
..text to number convs numeric characters to text strings
..NOTE: there is no number type so it cannot be declared like list, text, and file
..the third item in the list argument will be discarded
@conv({#var, number})
In the previous post, I mentioned that I am using Bash far less frequently as my work generally involves fewer server-related tasks and more business process type workflows. This generally means that my Javascript (by way of Google Apps Script) abilities have moved from "How do I do this?" to "How do I do this better!" In the past this led me to fleshing out the nuances of command line scripting (e.g. my post about reading files without cat), but JavaScript offers much deeper depths to plumb, many of which I have only discovered in passing.
One particular aspect of JavaScript objects has always bothered me: how do I use an object in a way that allows me to return a default value while still allowing access to properties contained within. Until today, I had assumed this wasn't possible, and I was trying to bring too much of my Bash-oriented brain to this relatively new-to-me langauge. However, looking through Stackoverflow I discovered a method that met my perhaps non-standard needs.
javascript
/*
Add object methods/values to a string while retaining the original value
from Stack Overflow: https://unforswearing.com/shorten/?KRn
*/
function addPropertiesToString() {
var str = 'hello';
// create string object
str = new String(str);
// add object properties
// repeat() is available per MDN, but not Google Apps Script
// see MDN: https://unforswearing.com/shorten/?ywk
str.repeat = function(x) {
var r = '';
for (var q = 0; q < x; q++) {
r = r + str;
}
return r;
};
// prepend text to str
str.prepend = function(text) {
var wpre = text + str;
return wpre;
}
// a different hello
str.alternate = 'howdy';
// Logs for the methods above:
Logger.log(str.repeat(3));
// [17-09-19 17:34:46:698 EDT] hellohellohello
Logger.log(str.prepend('someone said: '));
// [17-09-19 17:34:46:699 EDT] someone said: hello
Logger.log(str.alternate);
// [17-09-19 17:34:46:699 EDT] howdy
// And accessing the original str value
Logger.log(str);
// [17-09-19 17:34:46:700 EDT] hello
}
Having only recently explored objects, I wondered if setting a default string value after adding a property was possible:
javascript
function addPropertiesBeforeString() {
var newStr = new String();
newStr.goodbye = function() { return 'see ya!' };
Logger.log(newStr.goodbye());
// [17-09-19 17:34:46:700 EDT] see ya!
newStr = 'Greetings!'
Logger.log(newStr);
// [17-09-19 17:34:46:701 EDT] Greetings!
try {
// Doesn't work!
Logger.log(newStr.goodbye());
} catch(e) {
Logger.log(e);
// TypeError: Cannot find function goodbye
// in object Greetings!
}
}
Overall though, this opens up some interesting possibilities. I can think of at least one fairly practical example:
javascript
function stringMethodExample() {
var htmlMessage = '<h1>Hello</h1><br><br>Attached is a copy of your form responses.'
+ 'If you have any questions, get in touch!';
var file = DriveApp.getFileById(123456789);
htmlMessage = new String(htmlMessage);
htmlMessage.sendEmail = function(submitter, subject, body, htmlMessage) {
GmailApp.sendEmail(submitter, subject, body, {htmlBody: htmlMessage});
}
// Do other things that will generate the submitter and subject...
var submitter = 'you@yourdomain.com';
var subject = 'Form Responses';
// blank body because we are using htmlBody;
var body = '';
htmlMessage.sendEmail(submitter, subject, body, htmlMessage);
}
In the above example, logging htmlMessage by itself would return the original string, which could be used elsewhere in the script (as part of a generated PDF, for example).
While this doesn't save much time or increase productivity, it does keep the email scoped to the htmlMessage and with a few additional lines of code, an emailHtmlMessage class could be created to generate this type of object on the fly.
This was a fun exercise and my first real foray into deeper JavaScript methods/practices. Get in touch if you have any comments or suggestions!
I have been trying to find ways to avoid useless use of cat#Uselessuseofcat) but could never come up with a method of reading a file that did not involve creating an entire for or while loop. This problem crossed my mind yesterday and, after a quick google search, I found a neat way to avoid this in a single small command:
# contents of 'hi.txt' cat hi.txt hi hello ok
# reading hi.txt without 'cat' echo $(<hi.txt) hi hello ok
Unfortunately, the command does not preserve new lines but for those times when you only need to read from a single line file, this does the trick.
Looking through the bash oo framework repository I noticed the use of :: in function names. I have never seen this in bash scripts prior to using this framework, and I was curious about how I could leverage these in scripts to help more easily identify related processing commands.
Interestingly, some things work very well and some not so much:
bash
2:00 PM [~] > alias hi::there='echo hi'
2:00 PM [~] > hi::there
hi
2:00 PM [~] > hi
bash: hi: command not found
2:00 PM [~] > there
bash: there: command not found
2:00 PM [~] > alias hi.there='echo hi'
2:00 PM [~] > hi.there
hi
2:00 PM [~] > hi
bash: hi: command not found
2:01 PM [~] > there
bash: there: command not found
2:01 PM [~] > hi.there() { echo hi; }
bash: syntax error near unexpected token `(`
2:01 PM [~] > hi:there() { echo hi; }
2:01 PM [~] > hi:there
hi
2:01 PM [~] > hi
bash: hi: command not found
2:01 PM [~] > there
bash: there: command not found
A quick search for Bash Namespaces brings up a weath of additional information for using this technique.
Some lessons taken from "Unofficial Bash Strict Mode" placed into a simple strict function that can be called at will.
bash
#!/usr/bin/env bash
strict() {
set -Ceo pipefail
IFS=$'\n\t'
alias cp='cp -i'
alias mv='mv -i'
alias rm='rm -i'
error_exit() {
echo "$1" 1>&2
exit 1
}
_end_strict() {
unalias cp
unalias mv
unalias rm
set +Ceo
}
trap error_exit ERR
trap _end_strict ERR EXIT SIGHUP SIGTERM
}
I love recreating functionality found in non-free applications. The latest example of this is the New File menu app on Product Hunt. This application essentially provides the user with a method of creating new files on the fly, from any folder within finder. This is a wonderful tool that can be recreated with a little applescript. Open the Script Editor and enter the code below:
tell application "Finder"
set FoldName to selection as text
set FoldName to POSIX path of FoldName
set newfile to text returned of (display dialog "Enter file name and extension" default answer "")
set FileName to FoldName & newfile
set FileName to POSIX path of FileName
set FileName to quoted form of FileName
do shell script "touch " & FileName
do shell script "open " & FileName
end tell
Note: When using this script you must enter the file name and extension. Finder will automatically open the file in the default application. For example, creating "My Notes.txt" will open the new file in Text Edit (or your default application for .txt files).
To add the script to Finder, save as "New File" (without an extension) and click "cmd" while dragging the script from the folder to the Finder toolbar. You now have a quick shortcut to create a new file any time you need one, and you saved yourself a few bucks in the process.
My tool to get a list of links from Google Chrome has been updated more than a few times since the previous post, including the ability to save links to instapaper or pastebin, and an option to save the page(s) to pdf (via wkhtmltopdf). Here's the brand new full list of options:
-s to save the links to a file on the desktop
-c to copy the links to your clipboard
-v to print the links to stdout with leading text
-p to print the links to stdout
-i to save the link(s) to instapaper
-b to save the link(s) to pastebin.com
-w to save each url as a pdf (saves the page via 'wkhtmltopdf')
-h prints this help messageNote:
- lnks accepts one option. the program will fail if run with more than one option.
- using option -s will allow you to specify an output file, such as:
lnks -s searchterm matchinglinks.txt
I spent a while trying to figure out what I want to do next with lnks and feel pretty good about what I have planned:
Every now and again I take Safari out for a spin, and during one of those times I created surls to mimic lnks functionality. Merging surls with lnks makes a lot of sense, and got me thinking maybe I can add...
Most (if not all) of these browsers are scriptable via Applescript and it's gonna be great to brush up on my Applescripting. I am more than a bit out of practice.
I was super excited to add Instapaper support to lnks but got super bummed when I saw that they were sold to Pinterest. I probably won't drop Instapaper support for lnks (despite my silly threat), but I definitely want to add an alternative. Pinboard is a clean, well developed option I have on the horizon.
It'd be really great to not have to specify any new or old options every time lnks is run, so how about a configuration file? This will essentially be an expanded version of the current conf file that will be read at runtime.
There are a few other maintenance updates that I have planned for the far distant future, but I'm not quite there yet. Check back soon!
PROCESS:
* Files in the Originals folder are immediately processed by FFMPEG/HandbrakeCLI
* Files in the Originals folder that are 1 week old are sent to permanent storage* A script will alert me to new files, or
* The file will automatically be moved to a folder on my desktopISSUES:
* Personal account is too small
* Training account will put files on Jons computer
* Work account will work for anything under 2 GB
* Sub with Google Drive (up to 40 GB)?* Script did not wait until the video fully processed.
* That issue might be specific to applescript/hazel? +---------+
+-->|Originals+----+-------+
| +---------+ | | +----------------+
| | | |FFMPEG/HANDBRAKE+-----------+
| | | +----------------+ |
|raw exp | | ^ |
+------+ | | | | |
| | raw exp +-------------+ | v | |
| +-------->| | | +-----------+---+ |
| MAC | | DROPBOX | | | Processing PC | |
| |<--------+ | | +---------------+ |
| | pross'd +-------------+ | |
+------+ ^ | |
| | |
| | |
| | |
|processed |after 1 wk +--> +---------------------+ |
| +--------------->|Raw/Originals Storage| |
| +-------+ +---------------------+ |
+----+Exports| |
+-------+ |
^ |
| |
| <--------+compressed files |
+----------------------------------------------------+
Sources:
Navi Note Processing Language
Navi is a simple way to take and process notes. Take notes using the simple Navi syntax and when you need a review of the important tasks, low priority tasks, comments, and contexts, run it through the Navi processor.
SYNTAX
@Project - Normal priority/unprioritized task/item/comment/note > High priority task/item ∆ Backburner task/item (alt+j) † Waiting for something before continuing (alt+t) x Completed task/item ? Question about a task/project ! Important comment about a task/project $$ end of project
Parsing
sed -n '/@Project/ ,/$$/' grep '>' (etc)
The goal is to be fast and simple, without having to remember formatting rules. When run through the parser, all Ivan Markup is compiled into HTML, which can then be optionally converted to pdf via wkhtmltopdf. Ivan is built entirely with Ruby and Sed.
I built Ivan to suit my needs, and so it doesn't have a lot of the capabilities of other "lightweight markup languages." Specifically:
<h1> and <h2> header optionsSince this was a test of my limited programming capabilities, the syntax rules are pretty strict
=h1= ==h2== *bold* /italic/ _underline_ - list item - - list item - - nested list item - - nested list item -
Possible later additions:
;http://google.com; ;google;http://google.com; +http://google.com/smiley.gif+ +smiley+http://google.com/smiley.gif+ +;me@email.com;+ +;my email+;me@email.com;+ </code/> </ code block />
To start, write functions for each conversion that is needed: 1. h1 2. h2 3. bold 4. italic 5. underline 6. links w/o alt text 7. links w/ alt text 8. email w/o alt text 9. email w/ alt text 10. inline code
FUNCTIONs 1-10
DOCUMENT = ARGV[0]
OPTION = ARGV[1]
OPTION2 = ARGV[2]
Read file DOCUMENT
WORDS = words of DOCUMENT
if WORDS contains (markup syntax 1)
do FUNCTION 1
elsif WORDS contains (markup syntax 2)
do FUNCTION 2
(etc through 10 functions
Done
Case OPTION
when “-o”
save as newname.html
when “-f”
save as name.pdf,md
Done
Case OPTION2
when “-o”
save as newname.html
when “-f”
save as name.pdf,md
Done
The parser will read each line and then each word of the file, compare it to each of the 10 functions using ‘if’ statements in a ‘for’ loop. If it matches one, the function containing the code will process the word. If it doesn’t match any, it will continue to loop through the words in the file until there are no more words.
Ivan parser will exist as a command line only (because this has no place in Applescript). The script will take one file as an argument and convert to HTML by default. I am undecided about accepting processing options, but some ideas would be to
The internal Ivan parsing will be written by me. The pdf and md formats will happen via ‘wkhtmltopdf’ (ruby) and ‘pandoc’ respectively.
**Considerations**
- Possibly use the ‘++’ option to stop the script from starting at the beginning# Sed
## bold:
sed 's/ :/ <bold>/g' | sed 's/: /<\/bold> /g' | sed 's/^:/ <bold>/g' | sed 's/:$/<\/bold> /g' | sed 's/:./<\/bold>/'
## paragraph breaks:
sed 's/^/<p>/' | sed 's/$/<\/p>/'
# Awk
## get alt text for link
awk -F+ '{ print $1 }' | sed 's/{//'