2021
2020
2021

Nginx and Security in 2021

3 March 2021

I’ve made a few posts in the past about getting nginx up to scratch with security, specifically SSL’s. These are out of date now and pointless importing.

Pretty sure most admins that run nginx are already using it in stripped down chroot environment when security is a concern. However, SSL configurations are still getting regularly overlooked.

Check out the full configuration at the bottom if you are in a hurry.

This is especially common when using something like LetsEncrypt and certbot to just do the configuration automatically. Don’t get me wrong, it’s a really good utility to sort out an SSL quickly, but it has compatibility in mind and not security.

One of the most important things is which protocol to use. If you’re using nginx 1.13 or newer (mainline is well past this now) then you should be using TLSv1.3, otherwise you should be using at least TLSv1.2.

Note though, that in order to actually use TLSv1.3 (and not just have the option “enabled”), you need to compile nginx against a version of openssl that also supports it. CentOS 7 (and epel) do not currently support newer versions of openssl, and if you add the official nginx repos, you will still not have it (even with nginx-mainline).

Instead, you’ll need to compile it yourself, or use a repo like exove

Anyway, I’m not going to go in to detail about compiling nginx, or using repos, if you are trying to configure an nginx server, I’m assuming you have some experience with either or both of those processes.

At this point I’d also recommend enabling http2 protocol while we are here.

One thing you will need to do before enabling TLSv1.3 is create a Diffie-Hellman key, as you’ll need it for the ciphers we are going to use. This can be done with the following code:

openssl dhparam -out /etc/ssl/nginx/dh.pem 4096

This will take a bit of time on most systems, and you can adjust your config while you wait. It’s very likely you’ll still need to use TLSv1.2 for the time being, so we’ll leave that enabled for now, but feel free to disable it if you know you aren’t going to need it.

http {
  ....
  server {
    ....
    ssl_certificate /etc/ssl/nginx/fullchain.pem
    ssl_certificate_key /etc/ssl/nginx/private.pem
    ssl_dhparam /etc/ssl/nginx/dh.pem
    listen [::]:443 ssl http2 ipv6only=on;
    listen 443 ssl http2;
    ssl_protocols TLSv1.2 TLSv1.3;
    ....
          

Okay, great we’ve now enabled all the protocols we need, and we are well on our way to a decent set up.

The next part we’ll need to focus on is the ciphers and the algorithm curve used for the DH enabled ciphers (which is all of them).

http {
  ....
  server {
    ....
    ssl_ciphers ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384;
    ssl_ecdh_curve secp384r1;
    ....
          

This will enable only 5 ciphers. All of them being strong, and they are pretty well compatible with any newish browser. Android 7+, IE 11 (on Windows 10) and newer, Safari 9 and newer, Firefox 47+ and Chrome version 49+.

The weakest cipher is TLS_AES_128_GCM_SHA256, but even this is very strong, and really not worth worrying about removing (and adds a decent amount of compatibility).

The curve value is really quite complicated to explain, and I’m am definitely not someone who is fully up to speed with the technicalities of it. Plenty of others have done better jobs than I could, such as Andrea Corbellini

Anyway, these are cryptographically strong ciphers, and will likely be for a significant amount of time. So lets move on to the next stage.

Let’s setup the session values. This includes ticketing, timeout and cache timings.

http {
  ....
  server {
    ....
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    ssl_session_tickets off;
    ....
          

We want to avoid memory fragmentation, so we don’t use the builtin cache, and we want the cache to be shared across all worker processes for efficiency. the “SSL” portion of the cache line is just the name, and we are defining it as SSL for ease. As far as I can tell, this really doesn’t make any difference in security, but feel free to generate a random string if that’s more your thing.

The timeout simply states if a client can reuse the session, and for how long. 10 minutes is the recommended from every source I can find. Longer is unnecessary, and shorter has the potentially for breaking connections when using very limited bandwidth.

We also want to disable session tickets, which are client side and can cause vulnerabilities (such as ticketbleed). In almost all situations, they are unnecessary. You’ll know if you needed to use them, and you will probably not be reading this.

Okay! We are getting there. Now lets configure stapling, bear in mind you’ll need to make sure the intermediate certificates are included, either as part of your fullchain.pem or the older way of adding the CA bundle via the ssl_trusted_certificate directive.

http {
  ....
  server {
    ....
    ssl_stapling on;
    ssl_stapling_verify on;
    resolver 8.8.8.8 8.8.4.4 valid=300s;
    resolver_timeout 5s;
    ....
          

So a second part of stapling is verifying DNS via one or more nameservers, I’ve just included the Google nameservers here as they are reliable. I’d definitely suggest configuring the resolvers to be internal nameservers if you are deploying nginx on more than a single web server. We also set up how long verification from the resolvers will last, and the timeout of probing the nameservers.

The last part of the SSL configuration you’ll want to set up is adding the STS header.

http {
  ....
  server {
    ....
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
    ....
          

This sets the preload to 2 years. You’ll also need to submit a request to add your domain in to the preload list

I’d strongly advise reading through the documentation on that page before you do submit your domain though. As you wont be able to server non-SSL web traffic from the domain or any of it’s subdomains once it’s in the list.

Now, you should be able to restart nginx and test your site with something like SSLLabs or the Mozilla Observatory and you should see green across the board. Hurray!

So now your configuration file should look like this:

http {
  ....
  server {
    ....
    # Listening protocols
    listen [::]:443 ssl http2 ipv6only=on;
    listen 443 ssl http2;
    ssl_protocols TLSv1.2 TLSv1.3;
    # Certificate files
    ssl_certificate /etc/ssl/nginx/fullchain.pem
    ssl_certificate_key /etc/ssl/nginx/private.pem
    ssl_dhparam /etc/ssl/nginx/dh.pem
    # Ciphers to use
    ssl_ciphers ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384;
    ssl_ecdh_curve secp384r1;
    # Sessions
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    ssl_session_tickets off;
    # Stapling
    ssl_stapling on;
    ssl_stapling_verify on;
    resolver 8.8.8.8 8.8.4.4 valid=300s;
    resolver_timeout 5s;
    # Headers
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
    ....
          

Now, we could be done with this, but nginx is still, by default, vulnerable to other forms of attack. The good news is that the rest of the configuration is just setting more headers (except one). So let’s get started.

First, let’s stop some basic XSS vectors. And disable nginx version tokens from being sent.

http {
  ....
  server {
    ....
    # Disable nginx version tokens
    server_tokens off;
    # Headers
    ....
    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
          

All three of these headers tell the client that fundamentally we should be retrieving or sending data (like POST) between domains. And we will never use frames, of any kind.

There’s some new headers that have been recently introduced (at least since my last update on this topic).

Referrer-Policy, which determines what our website will do with referrer headers, I tend to just disable it entirely for privacy.

Expect-CT is how we tell clients that they shouldn’t even load our site if our certificate isn’t in the CT list

Feature-Policy only really applies to mobiles, but basically says what the client should expect our site to want to try and do. I disable everything, but if for whatever reason your site really wants access to the magnetrometer of the device, you should set this to something other than none.

http {
  ....
  server {
      ....
      # Headers
      ....
      add_header Referrer-Policy "no-referrer";
      add_header Expect-CT "enforce, max-age = 30, report-uri = 'https://report-uri.com/r/d/ct/enforce'";
      add_header Feature-Policy "geolocation 'none'; midi 'none'; notifications 'none'; push 'none'; sync-xhr 'none'; microphone 'none'; camera 'none'; magnetometer 'none'; gyroscope 'none'; speaker self;vibrate 'none'; fullscreen self;payment 'none'; ";
          

The final header is contentious, mostly with web apps, because coding practices are, frankly, diabolical. However, even if you have to set it to really permissive, you should be sending the Content-Security-Policy header. You might have to tweak this, a lot though.

http {
  ....
  server {
    ....
    # Headers
    ....
    add_header Content-Security-Policy "default-src 'none';"
          

This header WILL break most WordPress installs, so be very mindful of what you are doing with it. I would strongly advise reading Scott Thelme’s article on this.

And it will take a lot of trial and error to get it to work as you want it to and you probably wont be able to be as restrictive as you want to be with it (I know I’m not on this site). I’d suggest adding all the different directives and set them all to ‘none’ and work backwards from there. Chromium/Chrome’s debugger is very good with this, just look in the console (Ctrl+Alt+J) and see what is failing, and it will likely give you a solution.

And that’s it! Your nginx setup should be pretty well secured for SSL and XSS attack prevention. The final configuration should look like this:

http {
  ....
  server {
    ....
    # Listening protocols
    listen [::]:443 ssl http2 ipv6only=on;
    listen 443 ssl http2;
    ssl_protocols TLSv1.2 TLSv1.3;
    # Certificate files
    ssl_certificate /etc/ssl/nginx/fullchain.pem
    ssl_certificate_key /etc/ssl/nginx/private.pem
    ssl_dhparam /etc/ssl/nginx/dh.pem
    # Ciphers to use
    ssl_ciphers ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384;
    ssl_ecdh_curve secp384r1;
    # Sessions
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    ssl_session_tickets off;
    # Stapling
    ssl_stapling on;
    ssl_stapling_verify on;
    resolver 8.8.8.8 8.8.4.4 valid=300s;
    resolver_timeout 5s;
    # Disable nginx version tokens
    server_tokens off;
    # Headers
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header Referrer-Policy "no-referrer";
    add_header Expect-CT "enforce, max-age = 30, report-uri = 'https://report-uri.com/r/d/ct/enforce'";
    add_header Feature-Policy "geolocation 'none'; midi 'none'; notifications 'none'; push 'none'; sync-xhr 'none'; microphone 'none'; camera 'none'; magnetometer 'none'; gyroscope 'none'; speaker self;vibrate 'none'; fullscreen self;payment 'none'; ";
    add_header Content-Security-Policy "default-src 'none';"
    ....
          

Cisco ASA Firewall Management via Serial USB

20 January 2021

Had a bit of fun recently with a problematic cisco 5510 and needed to locally connect to it in the data centre. Figured I’d make a post about the exact steps needed to connect to it.

You’ll likely need a serial to USB adapter first and foremost. Then you’ll also need an RJ to serial cable.

Then, connect the RJ to the console port on the firewall.

Now, you can either use minicom or tio to connect. I prefer tio but I’ll go through both as the setup is essentially the same.

The most important thing to know before starting is what the USB device name is, the simplest way to find out is to check with dmesg.

dmesg | grep tty

You’ll then get an output along the lines of this

maisy@cerberus ~ $ sudo dmesg | grep tty
[    0.000000] console [tty0] enabled
[15982.172287] usb 1-1: pl2303 converter now attached to ttyUSB0
          

Now we know what the device is called we can try using tio or minicom to connect. From everything I’ve read a baud rate of 9600 is the most commonly working rate, though I didn’t actually have any issues with the default of 115200. The stopbits / databits are more important and I found only a value of 8 for databits worked, but this seems to be the default anyway.

Both tio and minicom need to be run as root / sudo privileges.

tio can be completely configured from the command line

tio /dev/ttyUSB0 -b 9600 -d 8 -s 1

minicom may be able to be configured all from the command line, but I wasn’t sure about some of the switches. Assuming the databits / stopbits are correct, you can just set the baudrate and device

minicom -D /dev/ttyUSB0 -b 9600

However, you may want to check directly in the minicom settings

minicom -s

You then want to go to “Serial port setup”, then “Bps/Par/Bits” and change stopbits to 1 and databits to 8-N-1. Once done, press escape until you are back at the menu and “Save setup as dfl”. You can then run the previous command again.

Once you’ve connected, you can admin the firewall as you would via ssh.

I might make a follow up post on the actual admin side of the firewall, but I’m not that well versed in it right now.


Bash Interactive CLI Menus

12 January 2021

This was initially just going to be an import of my old post about interactive menus, but I decided to review my old code for this (and also my much older code in the original post) and decided that I could do better instead.

So the original code was:

#!/bin/bash
menu_main=("Second_Menu" "Third_Menu" "Fourth_Menu" "Exit")
menu_second=("Fifth_Menu" "Main_Menu")
menu_third=("Main_Menu")
menu_fourth=("Main_Menu")
menu_fifth=("Second_Menu")
function __menu__ () {
  k=${#@}
  j=0
  l=(${@})
  [[ ! ${e} ]] && e=0
  clear
  while [[ $j -lt $k ]]; do
    [[ ${e} == ${j} ]] && echo -ne "\e[41m" || echo -ne "\e[0m"
    echo -n "["; echo -n "${l[$j]}"; echo -e "]\e[0m"
    j=$[${j}+1]
  done
  read -s -n 1 userin
  case ${userin} in
    "B") [[ ${e} -lt $((${k}-1)) ]] && e=$[${e}+1]; __menu__ ${l[@]} ;;
    "A") [[ ${e} -gt 0 ]] && e=$[${e}-1]; __menu__ ${l[@]} ;;
    "") s=${e} ;;
    *) __menu__ ${l[@]} ;;
  esac
  userin=""
}
function __menu_second__ () {
  unset e s
  __menu__ ${menu_second[@]}
  case ${s} in
    0) __menu_fifth__ ;;
    1) __menu_main__ ;;
  esac
}
function __menu_third__ () {
  unset e s
  __menu__ ${menu_third[@]}
  case ${s} in
    0) __menu_main__ ;;
  esac
}
function __menu_fourth__ () {
  unset e s
  __menu__ ${menu_fourth[@]}
  case ${s} in
    0) __menu_main__ ;;
  esac
}
function __menu_fifth__ () {
  unset e s
  __menu__ ${menu_fifth[@]}
  case ${s} in
    0) __menu_second__ ;;
  esac
}
function __menu_main__ () {
  unset e s
  __menu__ ${menu_main[@]}
  case ${s} in
    0) __menu_second__ ;;
    1) __menu_third__ ;;
    2) __menu_fourth__ ;;
    3) clear; exit 0 ;;
  esac
}
function __bootstrap__ () {
  clear
  __menu_main__
}
__bootstrap__
          

Part of my original post gives some insight into some of the limitations of this approach, so I was already aware of it being far from ideal.

There is a bit of hard coded formating in that (which I don’t like, but it will not doubt evolve over time), but this does exactly what I need. To use it, you need to do a bit of leg work (but much less so than repeating this over and over).

Something else to bear in mind — “A” and “B” pick up the up and down arrow keys no problem, but they also pick up shift + a and shift + b.

As you can no doubt tell, you still need to run your case statements on the end results — however, the menu itself is standardised so that the output is always 0-n depending on the menu titles you put in.

A big limitation at the moment is getting the menu title is based purely on arrays (which break spaces) so titles cannot contain spaces.

It’s also a bit limiting with the “refresh” in that it will always clear the screen — so you can’t echo before a command. However, this will be addressed pretty soon, I just wanted to get the initial example out there.

Not much changed really on my next iteration of the code, other than I added some better styling and ability to use page up / down.

function __menu__ () {
  k=${#@}
  j=0
  l=(${@})
  [[ ! ${e} ]] && e=0
  clear
  while [[ ${j} -lt ${k} ]]; do
    [[ ${e} == ${j} ]] && echo -ne "\e[0;30m\e[47m" || echo -ne "\e[0m"
    echo -n "[ "
    w=$(( 20 - ${#l[${j}]} ))
    [[ $(( ${w} % 2 )) -eq 1 ]] && echo -n " "
    w=$(( ${w} / 2 ))
    t=0
    while [[ ${t} -lt ${w} ]]; do
      echo -n " "
      t=$[${t}+1]
    done
    echo -n "${l[${j}]}"
    t=0
    while [[ ${t} -le ${w} ]]; do
      echo -n " "
      t=$[${t}+1]
    done
    echo -e " ]\e[0m"
    j=$[${j}+1]
  done
  read -s -n 1 userin
  case ${userin} in
    "B"|"6") [[ ${e} -lt $((${k}-1)) ]] && e=$[${e}+1]; __menu__ ${l[@]} ;;
    "A"|"5") [[ ${e} -gt 0 ]] && e=$[${e}-1]; __menu__ ${l[@]} ;;
    "") s=${e} ;;
    *) __menu__ ${l[@]} ;;
  esac
  userin=""
}
          

So now we move on to the newer code. And what I think is a much more elegant approach to it.

#!/bin/bash
## vmenu
## USAGE:               vmenu [MENU_ITEMS]
vmenu() {
  declare USER_INPUT
  declare -i SELECTED_ITEM RETURN_ITEM TOTAL_ITEMS i j
  SELECTED_ITEM=0
  RETURN_ITEM=-1
  TOTAL_ITEMS=$(( ${#@} - 1 ))
  ## Catch interupts and clear the menu
  trap '
    for (( i=TOTAL_ITEMS; i >= 0; i-- )); do
            printf "\033[1000D\033[K\033[1A"
    done
    printf "\033[?25h"
    exit 1
  ' SIGINT
  ## Hide the cursor
  printf "\033[?25l"
  while (( RETURN_ITEM == -1 )); do
    ## Print the menu
    i=0
    for MENU_ITEM in "${@}"; do
      (( i == SELECTED_ITEM )) && printf "\e[0;30m\e[47m%s\e[0m" "$MENU_ITEM" || printf "%s" "$MENU_ITEM"
      ## Clearing line work around for leaving remains of previous menu on same line
      for (( j=0; j < 20; j++ )); do
        printf " "
      done
      printf "\n"
      (( i++ ))
    done
    ## Get arrow key input
    read -r -s -n 1 USER_INPUT
    case $USER_INPUT in
      'B'|'6') (( SELECTED_ITEM < TOTAL_ITEMS )) && (( SELECTED_ITEM++ )) ;;
      'A'|'5') (( SELECTED_ITEM > 0 )) && (( SELECTED_ITEM-- )) ;;
      '') RETURN_ITEM=$SELECTED_ITEM ;;
    esac
    ## Clear extra lines
    while (( i > 0 )); do
      printf "\033[1000D\033[K\033[1A"
      (( i-- ))
    done
  done
  ## Unhide the cursor and return menu entry
  printf "\033[?25h"
  return $RETURN_ITEM
}
vmenu_level_one() {
  vmenu "Go Deep" "Exit"
  case $? in
    0) vmenu_level_two ;;
    1) exit 0 ;;
  esac
}
vmenu_level_two() {
  vmenu "Go Deeper" "Back up" "Reload this one" "Exit"
  case $? in
    0) vmenu_level_three ;;
    1) vmenu_level_one ;;
    2) vmenu_level_two ;;
    3) exit 0 ;;
  esac
}
vmenu_level_three() {
  vmenu "Go Even Deeper" "Back up" "Exit"
  case $? in
    0) printf "\033[1000D\033[KToo deep!\n" ;;
    1) vmenu_level_two ;;
    2) exit 0 ;;
  esac
}
vmenu_level_one
          

So one of the biggest differences in this new menu is it doesn’t use clear, at all. It also cleanly removes the whole menu when you quit. I also added a trap in case the user CTRL+C’s it.

The other biggest difference is it also has fewer while loops and doesn’t rely on recursiveness to reexecute the function on every key press. Instead it just keeps going until the RETURN_ITEM value is valid.

And the final significant difference is that it uses return codes instead of an arbitrary variable ($s) that also needs to be global for it to work. This is really handy and if you ever wanted to make the case statement somewhere else, you could simply put the results of $? into another variable yourself.

I also decided that the horizontal menu needed revisions too, but the old code is almost the same as the vertical menu, and really not worth mentioning.

However, the new code for horizontal menus is significantly different.

#!/bin/bash
## hmenu
## USAGE:               hmenu [MENU_ITEMS]
hmenu() {
  declare USER_INPUT
  declare -i SELECTED_ITEM RETURN_ITEM TOTAL_ITEMS i
  SELECTED_ITEM=0
  RETURN_ITEM=-1
  TOTAL_ITEMS=$(( ${#@} - 1 ))
  ## Catch interupts and clear the menu
  trap '
    printf "\033[1000D\033[K\033[?25h"
    exit 1
  ' SIGINT
  ## Hide the cursor
  printf "\033[?25l"
  while (( RETURN_ITEM == -1 )); do
    ## Print the menu
    i=0
    for MENU_ITEM in "${@}"; do
      (( i == SELECTED_ITEM )) && printf "\e[0;30m\e[47m%s\e[0m" "$MENU_ITEM" || printf "%s" "$MENU_ITEM"
      (( i < TOTAL_ITEMS )) && printf " | "
      (( i++ ))
    done
    read -r -s -n 1 USER_INPUT
    case $USER_INPUT in
      'C') (( SELECTED_ITEM < TOTAL_ITEMS )) && (( SELECTED_ITEM++ )) ;;
      'D') (( SELECTED_ITEM > 0 )) && (( SELECTED_ITEM-- )) ;;
      '') RETURN_ITEM=$SELECTED_ITEM ;;
    esac
    printf "\033[1000D\033[K"
  done
  ## Unhide the cursor and return menu entry
  printf "\033[?25h"
  return $RETURN_ITEM
}
hmenu_level_one() {
  hmenu "Go Deep" "Exit"
  case $? in
    0) hmenu_level_two ;;
    1) exit 0 ;;
  esac
}
hmenu_level_two() {
  hmenu "Go Deeper" "Back up" "Reload this one" "Exit"
  case $? in
    0) hmenu_level_three ;;
    1) hmenu_level_one ;;
    2) hmenu_level_two ;;
    3) exit 0 ;;
  esac
}
hmenu_level_three() {
  hmenu "Go Even Deeper" "Back up" "Exit"
  case $? in
    0) printf "Too deep!\n" ;;
    1) hmenu_level_two ;;
    2) exit 0 ;;
  esac
}
hmenu_level_one
          

This works exactly like my new vertical menu, but is even more simplified because it doesn’t require working out how many lines there are.

2020

IPMI Web API

5 November 2020

So I’d been working on getting some IPMI tool intergration with a basic web interface and the easiest way to achieve that was to use a shell script that the PHP script would call.

This allowed me to do a lot of the processing through bash and required minimal amounts of PHP.

The script runs using a preset list of server config files in the “servers/” directory (relative to the script) each of which have the following:

hostname=servername
ip=1.1.1.1
username=root
password=password
lantype=lan
ssh=no
          

Most of that should be pretty self explanitory, the hostname allows you to specify a “nice name” for the server (used by the web front end), the IP is that the script will use to connect, the username and password allow you to issue commands without having to type out the login every time.

The lantype is either “lan” or “lanplus” depending on what the motherboard supports (read the ipmi documentation for more info).

The final entry is wether to connect via ssh or not (using the hard coded remote ssh host — this may change to a config file setting at some point, if the need arises).

When you have some server.cfg files in there, you can run “./ipmitools.sh –server SERVER COMMANDS” where SERVER is the file name (minus the .cfg part) and COMMANDS are the extra switches.

I haven’t yet got to the point of adding a help file at the moment, but the current commands (if you can’t deduce them from the script below) are:

--info full / power
--power on / off / cycle
          

Power info reports back a 1 for on, a 0 for off and a 2 for unreachable (the reason for these numbers rather than just outputting the IMPI error is due to simplicity in my Ajax checks).

The full shell script is here — if you plan on using SSH, you need to edit the ssh line in config:

#!/bin/bash
## Error reporting ##
function __error__ () {
  case ${1} in
    "config") echo "Malformed config file" ${2} "on line" ${3}; exit 1 ;;
  esac
}
#####################
## Read config file ##
function __config__ () {
  c="servers/${1}.cfg"
  i=1
  while read line; do
    [[ ${line} =~ ^(.*)[=](.*)$ ]] && var=${BASH_REMATCH[1]} && val=${BASH_REMATCH[2]} || __error__ "config" ${c} ${i}
    case ${var} in
      "hostname") hostname=${val} ;;
      "ip") ip=${val} ;;
      "username") username=${val} ;;
      "password") password=${val} ;;
      "lantype") lantype=${val} ;;
      "ssh") [[ ${val} == "yes" || ${val} == "1" ]] && ssh="ssh REMOTE@SSH.SERVER" || ssh="" ;;
    esac
    i=[${i}+1]
  done < ${c}
}
######################
## Get switches ##
function __switches__ () {
  while [[ ${@} ]]; do
    case ${1} in
      "--server") server=${2}; shift 2 ;;
      "--info")
        case ${2} in
          "full") command="full" ;;
          "power") command="power" ;;
          *) echo "Unknown operator ${2}"; exit 1 ;;
        esac
        shift 2
        ipmicommand="info"
      ;;
      "--power")
        case ${2} in
          "on") command="power"; extracommand="on" ;;
          "off") command="power"; extracommand="off" ;;
          "cycle") command="power"; extracommand="cycle" ;;
          *) echo "Unknown operator ${2}"; exit 1 ;;
        esac
        shift 2
        ipmicommand="command"
      ;;
      *) echo "Unknown switch ${1}"; exit 1 ;;
    esac
  done
}
##################
## Pull information ##
# info type (internal command), lantype, IP, username, password
function __ipmiinfo__ () {
  case ${1} in
    "full")
      command=`${6} ipmitool -I ${2} -H ${3} -U ${4} -P "${5}" chassis status`
      echo -e "${command}"
      exit 0
    ;;
    "power")
      command=`${6} ipmitool -I ${2} -H ${3} -U ${4} -P "${5}" chassis power status 2>/dev/null`
      if [[ ${command} == "Chassis Power is on" ]]; then
        echo "1"
      elif [[ ${command} == "Chassis Power is off" ]]; then
        echo "0"
      else
        echo "2"
      fi
      exit 0
    ;;
  esac
}
###########################
## Send Commands ##
# command type, lantype, IP, username, password, extra command
function __ipmicmd__ () {
  case ${1} in
    "power")
      command=`${7} ipmitool -I ${2} -H ${3} -U ${4} -P "${5}" power ${6}`
      exit 0
    ;;
  esac
}
#########################
## Startup ##
function __bootstrap__ () {
  __switches__ ${@}
  __config__ ${server}
  case ${ipmicommand} in
    "info") __ipmiinfo__ ${command} ${lantype} ${ip} ${username} ${password} "${ssh}"; exit 0 ;;
    "command") __ipmicmd__ ${command} ${lantype} ${ip} ${username} ${password} ${extracommand} "${ssh}"; exit 0 ;;
  esac
}
__bootstrap__ ${@}
          

I’m sure you can probably work out the PHP code to set this up, but the main bulk was the Ajax calls to run each command. I am in no way proficient with javascript but I managed to work out the following:

function loadServerPower(server){
  var xmlHttp;
  var prevState = document.getElementById(server).className;
  xmlHttp=new XMLHttpRequest();
  xmlHttp.onreadystatechange=function(){
    if(prevState == ""){
      document.getElementById(server).className = "serverItemX";
      document.getElementById("power-"+server).innerHTML="Connecting";
      document.getElementById("powercycle-"+server).disabled = true;
      document.getElementById("poweroff-"+server).disabled = true;
      document.getElementById("poweron-"+server).disabled = true;
    }
    if(xmlHttp.readyState==4){
      if(xmlHttp.responseText == "1" && prevState != "serverItem1"){
        document.getElementById(server).className = "serverItem1";
        document.getElementById("power-"+server).innerHTML="On";
        document.getElementById("powercycle-"+server).disabled = false;
        document.getElementById("poweroff-"+server).disabled = false;
      }else if(xmlHttp.responseText == "0" && prevState !="serverItem0"){
        document.getElementById(server).className = "serverItem0";
        document.getElementById("power-"+server).innerHTML="Off";
        document.getElementById("powercycle-"+server).disabled = false;
        document.getElementById("poweron-"+server).disabled = false;
      }else if(xmlHttp.responseText == "2" && prevState !="serverItem2"){
        document.getElementById(server).className = "serverItem2";
        document.getElementById("power-"+server).innerHTML="Unreachable";
      }
    }
  }
  url="?power="+server;
  xmlHttp.open("GET",url,true);
  xmlHttp.send(null);
}
function loadServerStatus(server){
  var xmlHttp;
  xmlHttp=new XMLHttpRequest();
  xmlHttp.onreadystatechange=function(){
    if(xmlHttp.readyState==4){
      document.getElementById("status-"+server).innerHTML=xmlHttp.responseText;
    }
  }
  url="?status="+server;
  xmlHttp.open("GET",url,true);
  xmlHttp.send(null);
}
function sendServerPower(server,power){
  if(power == 'powercycle'){
    var warning='Are you sure you want to power cycle '+server+'?';
  }else if(power == 'poweron'){
    var warning='Are you sure you want to power on '+server+'?';
  }else if(power == 'poweroff'){
    var warning='Are you sure you want to power off '+server+'?';
  }
  var response = confirm(warning);
  if(response){
    var xmlHttp;
    xmlHttp=new XMLHttpRequest();
    xmlHttp.onreadystatechange=function(){
      if(xmlHttp.readyState==4){
        document.getElementById(server).className = "serverItemX";
        document.getElementById("power-"+server).innerHTML="Acting";
        document.getElementById("powercycle-"+server).disabled = true;
        document.getElementById("poweroff-"+server).disabled = true;
        document.getElementById("poweron-"+server).disabled = true;
      }
    }
    url="?command="+power+"&server="+server;
    xmlHttp.open("GET",url,true);
    xmlHttp.send(null);
  }
}
function toggleServerStatus(id){
  var e = document.getElementById(id);
  if(e.style.display == 'block') {
    e.style.display = 'none';
  }else{
    e.style.display = 'block';
  }
}
          

I just ran a simple PHP loop on the directory contents of “servers/” to work out how many rows there shold be and applied the above javascript to buttons and information panels. And used GET switches to work out wihich command type was being called.

As usual, I’m not responsible for what you do with this information / scripts and will definitely not be held accountable for you turning off a live production server (I suggest you have a test server or, at the very least, read the IPMI documentation).


PHP Display Random Image

22 October 2020

This is a modified version of an uncredited script I found; it picks an image at random from a folder specified and displays them, works for different sized images as well.

Useful for things like forum signatures or site headers.

<?php
$folder = '.';  // Images location
$extList = array();     // Image types
$extList['gif'] = 'image/gif';
$extList['jpg'] = 'image/jpeg';
$extList['jpeg'] = 'image/jpeg';
$extList['png'] = 'image/png';
$img = null;
if (substr($folder,-1) != '/') {
  $folder = $folder.'/';  // Prevents remote folder setting
}
if(isset($_GET['img'])){
  $imageInfo = pathinfo($_GET['img']);
  if(isset($extList[strtolower($imageInfo['extension'])]) && file_exists($folder.$imageInfo['basename'])){
    $img = $folder.$imageInfo['basename'];
  }
}else{
  $fileList = array();
  $handle = opendir($folder);
  while (false !== ($file = readdir($handle))){
    $file_info = pathinfo($file);
    if (isset($extList[strtolower($file_info['extension'])])){
      $fileList[] = $file;
    }
  }
  closedir($handle);
  if(count($fileList) > 0){
    $imageNumber = time() % count($fileList);
    $img = $folder.$fileList[$imageNumber];
  }
}
if($img!=null){
  $imageInfo = pathinfo($img);
  $contentType = 'Content-type: '.$extList[$imageInfo['extension']];
  header ($contentType);
  readfile($img);
}else{
  if(function_exists('imagecreate')){
    header ("Content-type: image/png");     // Create a PNG as the output
    $im = @imagecreate (100, 100) or die ("Cannot initialize new GD image stream");
    $background_color = imagecolorallocate ($im, 255, 255, 255);
    $text_color = imagecolorallocate ($im, 0,0,0);
    imagestring ($im, 2, 5, 5,  "IMAGE ERROR", $text_color);
    imagepng ($im);
    imagedestroy($im);
  }
}
?>
          

Does anyone still use forums now? I’m not even sure they did that much in 2012 when I first wrote this…


cPanel MySQL Bulk Restores

17 September 2020

I had to do some serious hacking of code in a previous job, when innoDB messed up big time.

We have a remote backup server which takes the CPBackup tar files just for databases (though the whole directory structure is still there), but this presents a fairly irritating problem of requiring to untar each archive, find the database and move on. This also causes a lot of disk space to be consumed for no reason (especially when dealing with lots of databases in one account – not all the databases will have innoDB tables).

So anyway, I had to rush to code something that worked and ended up with a pair of scripts that take a list of databases to restore and does all the heavy lifting for you.

You’ll need to generate a list of databases you want to restore and stick it in the file listed as the variable “backupfile” in the scripts.

On the backup server you need to use “mysqlBackup” (I couldn’t think of a better name).

#!/bin/bash

basedir="/backups/thedate/"
backupdir="mysqlbackup"
backupfile="mysqlbackup.txt"
remoteserver="someserver.domain.tld"
remotedir="/root/"

if [ -d $basedir/$backupdir ]; then
  echo "Backup directory already created, skipping."
else
  mkdir $basedir/$backupdir
fi

while read line; do
  name=${line%%_*}
  db=${line#*_}
  dbfile=$name"_"$db
  echo $dbfile

  if [ -f "$basedir/$backupdir/$dbfile.sql" ]; then
    echo "Database copy already done, skipping."
  else
    cd "$basedir/$name"
    tar xvf cpmove-$name.tar.gz "cpmove-$name/mysql/$dbfile.sql"
    cd "cpmove-$name/mysql"
    cp "$dbfile.sql" "$basedir/$backupdir/"
    cd "$basedir/$name"
    rm -rf cpmove-$name/
    cd "$basedir/"
 fi

done < $backupfile

cd "$basedir/$backupdir"
tar -acf "$backupdir.tar.gz" *
rsync -P "$backupdir.tar.gz" root@$remoteserver:$remotedir
rsync -P "$backupfile" root@$remoteserver:$remotedir

exit 0
          

That will extract just the needed databases from the archive (so it can loop round the same archive several times without causing any unneeded overhead), move them to a temporary location, tar all the databases up and send both the new tar file and the database list off to the server you are restoring on (a MySQL server or just a normal cPanel server).

Then, on the actual server holding the live databases, you’ll want to use “mysqlRestore”

#!/bin/bash

basedir="/root/"
backupdir="mysqlbackup"
backupfile="mysqlbackup.txt"
errLog="mysqlRestore.err"

if [ -d "$basedir/$backupdir" ]; then
  echo "Backup directory already created, skipping."
else
  mkdir "$basedir/$backupdir"
fi

if [ -f "$basedir/$backupdir.tar" ]; then
  mv $backupdir.tar "$basedir/$backupdir/"
  cd "$basedir/$backupdir/"
  tar xvf $backupdir.tar
  rm $backupdir.tar
  cd "$basedir/"
else
  echo "Backup tar does not exist, hoping for the best and skipping."
fi

while read line; do
  name=${line%%_*}
  db=${line#*_}
  dbfile=$name"_"$db
  echo $dbfile

  if [ -f "$basedir/$backupdir/$dbfile.sql" ]; then
    echo $dbfile
    mysql -e "create database $dbfile;"
    mysql $dbfile < "$basedir/$backupdir/$dbfile.sql"
  else
    echo "SQL Backup file does not exist, skipping."
    echo "$dbfile.sql" >> $errLog
  fi

done < $backupfile
exit 0
          

You need to be running as a user with a my.cnf defined so you can log in to your MySQL root without a password, otherwise it will be very tedious.

All it does is extracts the archive into a new temporary folder then imports each database one by one.

It also runs pretty quick when doing this and I’ve not seen any issues when restoring several hundred in one go (yet, anyway).

These scripts are totally work in progress though, but thought it might help out with people who have serious database problems and just want to do a restore to the last backup – you could easily modify the scripts to run from just a separate partition as well if that’s how your server(s) are configured.

Also, I’m not responsible if you break something using these!


DD-WRT IP Routing

24 August 2020

I have a Linksys router and, naturally, I’ve got custom firmware on it.

I went for DD-WRT quite some time ago and I’ve gotten used to it. One of the best features is the ability to manage everything via SSH.

However, the biggest hurdle with doing that is getting to grips with the limitations of it’s shell (which is ash — not bash). Things that you would take for granted in bash, such as arrays, are not necessarily present in ash.

Function declaration is much stricter as well — requiring the () to be immediately after the function name with no spacing. You also can’t use the (albeit unnecessary) “function” declaration either.

Anyway — I had a need to route multiple external IP addresses to an internal machine and rather than doing the iptables commands every time, I thought I would write a script to do it for me.

#!/bin/sh
## Usage: __port_route__ [IPADDRESS] [EXTERNAL_PORT->INTERNAL_PORT : EXTERNAL_PORT->INTERNAL_PORT] ##
__port_route__() {
  ipaddress="${1}"
  ports="${2}"
  oldIFS=${IFS}
  IFS=" : "
  for route in ${ports}; do
    IFS="->"
    local i=0
    for port in ${route}; do
      if [ ${i} == 0 ]; then
        local extport=${port}
      else
        local intport=${port}
      fi
    local i=1
    done
    iptables -t nat -I PREROUTING -p tcp --dport ${extport} -j DNAT --to ${ipaddress}:${intport}
    iptables -I FORWARD -p tcp -d ${ipaddress} --dport ${extport} -j ACCEPT
    echo "External port ${extport} routed to ${ipaddress}:${intport}"
  done
  IFS=${oldIFS}
}
## Fair warning -- port routing *MUST* be enclosed in quotes, preferably single quotes. otherwise you'll end up creating files. ##
__port_route__ ${1} ${2}
          

Usage:

portroute 255.255.255.255 '80->80 : 22->22 : 143->143'

I had to go round the houses to get the, potentially, different external and internal ports in as different variables so I could use them all in one go — however, I think considering the restrictions that the result is fairly clean.

Also, the ports and IP address mentioned in the script are not the ones I had used (they are all blocked again now anyway).

As usual, I’m not responsible for what you do with this script or if it breaks anything.

Note I actually changed my router a few months ago, and I no longer use DD-WRT, mainly because the speed of a/b/g wasn’t fast enough to match my actual internet speed, and I have n capable laptops and phone now.

Instead, I’m using a Turris Omnia which is, frankly, fucking amazing. I’ll do an actual post on it at some point in the future.


Resource Usage in Bash

19 August 2020

So I stumbled across some fairly useful commands recently and decided to expand things out a bit.

ps awwfux | grep --color=none $USER | less -S  

This command will print the processes running as the current user (and the login process for that user), and display them in a tree structure. The less command allows you to scroll both horizontally and vertically. I also love pronouncing the ps arguments phonetically.

ps -eo pcpu | egrep -v "0.0|CPU" | awk '{sum+=$1}END{print sum}'

This returns the current CPU usage (as a percentage); it doesn’t necessarily have much of a use on it’s own, but with a little bit of work you could do something like this:

!/bin/bash
cpu_usage() {
  p=$( ps -eo pcpu | egrep -v "0.0|CPU" | awk '{sum+=$1}END{print sum}' )
  p=$( echo "${p} * 10" | bc )
  p=$(( ${p:0:2} / 10 / 2 ))
  i=0
  printf "["
  while [[ ${i} -le 50 ]]; do
    [[ ${i} -le ${p} ]] && printf "#" || printf "-"
    (( i++ ))
  done
  printf "]"
}
main() {
  cpu_usage
  sleep 1
  for (( i=0; i <= 102; i++ )); do
    printf "\b"
  done
  main
}
main
          

Which will display a nice meter:

[#####----------------------------------------------]

Similarly, with a bit of effort you can do the same for memory usage. Starting off with:

t=$( echo $( cat /proc/meminfo | grep "MemTotal" | awk '{print $2}' ) / 1024 | bc )
f=$( echo $( cat /proc/meminfo | grep "MemFree" | awk '{print $2}') / 1024 | bc )
u=$(( $t - $f ))
p=$( echo "scale=2
${u} / ${t}" | bc | cut -d "." -f 2 )
echo ${p}
          

Which outputs a similar result to the initial CPU usage command. But can just as easily be converted into a bar like this:

#!/bin/bash
mem_usage() {
  t=$( echo $( cat /proc/meminfo | grep "MemTotal" | awk '{print $2}' ) / 1024 | bc )
  f=$( echo $( cat /proc/meminfo | grep "MemFree" | awk '{print $2}') / 1024 | bc )
  u=$(( $t - $f ))
  p=$( echo "scale=2; ${u} / ${t}" | bc | cut -d "." -f 2 )
  i=0
  printf "["
  while [[ ${i} -le 50 ]]; do
    [[ ${i} -le $(( ${p} / 2 )) ]] && printf "#" || printf "-"
    (( i++ ))
  done
  printf "]"
}
main() {
  mem_usage
  sleep 1
  for (( i=0; i <= 102; i++ )); do
    printf "\b"
  done
  main
}
main
          

With the familiar bar output.

As you can probably tell, the code is almost identical, the main difference is what generates ${p}. Also, it’s really pretty simple to change the size of the bars, you just have to change the while loop to a lower number (this is the number of characters the bar will take up) and divide ${p} by 100 divided by whatever the value of ${i} will reach. EDIT: So, I had a quick think, and it’s again trivial to work the percentage out for disk usage as well. Fortunately, you don’t even need to do much in the way of calculations if you already have “df”; you can simply run:

df -h --output=source,pcent | grep "/dev/" | awk '{sum+=$2}END{print sum}'

You don’t need the “source” output field, but it makes it easier to distinguish physical drives versus RAM disks etc. Again, can be made into a fancy bar with minimal effort:

disk_usage() {
  p=$( df -h --output=source,pcent | grep "/dev/" | awk '{sum+=$2}END{print sum}' )
  i=0
  printf "["
  while [[ ${i} -le 50 ]]; do
    [[ ${i} -le $(( ${p} / 2 )) ]] && printf "#" || printf "-"
    (( i++ ))
  done
  printf "]"
}
main() {
  disk_usage
  sleep 1
  for (( i=0; i <= 102; i++ )); do
    printf "\b"
  done
  main
}
main
          

Then, if you wanted all three in one, you could easily put the meters in like:

main() {
  cpu_usage
  printf "\t"
  mem_usage
  printf "\t"
  disk_usage
  sleep 1
  for (( i=0; i <= 312; i++ )); do
    printf "\b"
  done
  main
}
          

Though using the “\b” (backspace) character is very useful in keeping the terminal clean, it is limited when it comes to line breaks (in that it doesn’t work), so using “clear” may be a better option if you were building some kind of GUI. Alternatively, you could plug the numbers into dialog / xdialog really easily. Something like this would work (although doesn’t provide any options and you need to * 10 / 10 to round the number):

dialog --gauge test 10 30 $(( $( ps -eo pcpu | egrep -v "0.0|CPU" | awk '{sum+=$1}END{print sum}' ) * 10 / 10 ))
          

C Line Editor

14 June 2020

While most of the stuff I’ve done in C has been trival at best (a very basic calculator and a basic temperature converter [that does Kelvin, Rankine, Delisle and Newton]), I had also worked a fair amount on a line based text editor.

Because I am still new to C, doing a full, visual, character by character editor would have probably been a bit much. So I opted for an “ed” inspired editor which, while vastly less powerful than ed, seems to work pretty well.

I came up with some really random internal shortcuts for it, but the general usage is fairly simple. I managed to total around 750-1000 lines of code for it and I’m sure there are ways of improving, but still… I like it!

So, because pasting all the code here is a little ridiculous, I’ll just list the highlights.

Firstly, the prototype functions:

void clear_string ( char s[BUFSIZ] );
void error ( int i );
void check_return ( int i );
int show_help ( void );
int command_file_temp_create ( char f[BUFSIZ] );
int command_file_temp_remove ( char f[BUFSIZ] );
int command_file_temp_save ( char f[BUFSIZ] );
int command_internal_line_count ( char f[BUFSIZ] );
int command_internal_list_all ( char f[BUFSIZ] );
int command_internal_list_all_silent ( char f[BUFSIZ] );
int command_internal_list_single ( char u[BUFSIZ], char f[BUFSIZ] );
int command_internal_write_all ( char u[BUFSIZ], char f[BUFSIZ] );
int command_internal_write_single ( char u[BUFSIZ], char f[BUFSIZ] );
int command_internal_purge_all ( char f[BUFSIZ] );
int command_internal_purge_single ( char u[BUFSIZ], char f[BUFSIZ] );
int command_internal_shell_passthru ( char u[BUFSIZ] );
int command_internal_discard ( char f[BUFSIZ] );
int command_internal_save ( char f[BUFSIZ] );
int command_internal_delete ( char f[BUFSIZ] );
int command_internal_append_single ( char u[BUFSIZ], char f[BUFSIZ] );
int command_internal_search_keyword ( char u[BUFSIZ], char f[BUFSIZ] );
int command_internal_search_keyword_count ( char u[BUFSIZ], char f[BUFSIZ] );
          

That should give you an idea of the posibilities avaliable within the tool itself (the help screen is obviously more effective).

One of the functions I really like is the internal_search_keyword function. You specify a word and the function lists all the lines that the word appear on (colours / styles are not yet included, so there’s no way of “bolding” the keyword).

int command_internal_search_keyword ( char u[BUFSIZ], char f[BUFSIZ] ) {
FILE *temp;
char file_temp[BUFSIZ];
char line[BUFSIZ];
char word[BUFSIZ];
char d[BUFSIZ], e[BUFSIZ];
int i = 1, j = 0, c = 0, x = 0, l, k, m, n;
char *ptr;
clear_string ( file_temp );
clear_string ( line );
clear_string ( u );
clear_string ( word );
strcat ( file_temp, f );
strcat ( file_temp, ".tmp" );
l = command_internal_line_count ( f );
clear_string ( d );
sprintf ( d, "%d", l );
k = strlen ( d );
scanf ( "%[^\t\n]", u );
while ( getchar () != ( int ) '\n' );
if ( u[0] == '\0' ) {
  x = 2;
} else {
  while ( u[c] != '\0' ) {
          u[c] = u[c+1];
          c++;
  }
  temp = fopen ( file_temp, "r" );
  if ( temp == NULL ) {
          x = 1;
  } else {
    while ( fgets ( line, BUFSIZ, temp ) ) {
      if ( ( ptr = strstr ( line, u ) ) != NULL ) {
        clear_string ( e );
        sprintf ( e, "%d", i );
        n = strlen ( e );
        m = k - n;
        switch ( m ) {
          case 1:
            printf ( "0" );
            break;
          case 2:
            printf ( "00" );
            break;
          case 3:
            printf ( "000" );
            break;
          case 4:
            printf ( "0000" );
            break;
          case 5:
            printf ( "00000" );
            break;
          case 6:
            printf ( "000000" );
            break;
          default:
            break;
        }
        j++;
        printf ( "%i|%s", i, line );
      }
      i++;
    }
    if ( fclose ( temp ) != 0 ) {
      x = 1;
    }
  }
}
return ( x );
}
          

The thing I really like about this function is the line numbering aligning itself to the total line count of the file. It’s not elegant, but it feels much nicer when working with large files.

I’ve also included that simple switch on printing the full file as well, so you get a better feel for where things are.

Here’s the help screen as well, which shows you the possibilities:

%% Print total lines
&& Print all line content
& [LN] Print [LN] line content
&&&    Print all line content with line numbers
^^ Write new lines from start of file
^ [LN] Write new lines from [LN]
$\$ Finish writing new lines
-- Remove all lines from file
- [LN] Remove single line from file
+ [LN] Add a single new line after [LN]
// [K] Search for [K] in file and return lines
/ [K]  Search for [K] in file and return count
.. Save changes to file and quit
.  Save changes to file
,, Discard unsaved changes and quit
,  Discard unsaved changes and reload last save
<< Discard changes, delete file and exit
!! Command line passthrough
## Print version number
?? Show this help screen
          

You can find the full source on the “cede” project on my git page.


Bash Game Programming

23 May 2020

I decided that I would try and build a very simple game engine in Bash. The premise was a bit ridiculous, but Bash is more flexible than people often give it credit for. Sure, it’s not efficient; it’s not pretty. But it does get the job done… More or less.

I’m a long way off completion yet, but I already have some of the fundamentals sorted. The main ones being map generation and player movement.

For now it does not generate random levels (which would be far more rouge like); however, it is completely capable of interpreting what each of the (limited) characters in the board file do. The main generation loop looks like this:

function board_draw () {
  declare -i i j
  i=0
  j=0
  while read -n1 c; do
    [[ ${i} -eq ${width} ]] && i=0 && j=$[${j}+1] && printf "\n"
    printf "%c" "${c}"
    [[ "${c}" == "${nextdoor}" ]] && py[${map}]=${j} && px[${map}]=${i}
    [[ "${c}" == "${prevdoor}" ]] && cy[${map}]=${j} && cx[${map}]=${i}
    m[${j}]="${m[${j}]}${c}"
    (( i++ ))
  done < "${base}/maps/${map}.map"
  tput cup $(( ${height} + 5 )) 0
  printf "Use IJKL to move. Use 'q' to quit."
  return 0
}
          

In actual fact, the map drawing doesn’t do collision detection, but instead inserts each line into an array (m[]). You can then check m[LINE] for character X to determine the positions of things.

It also registers it’s next and previous door locations into py[], px[] and cy[], cx[]. This allows a smoother transition between levels without requiring position 1:1 to always be used.

The actual size of the map is defined in the main loop — though this may be changed at a later date to allow for different dimensions of levels.

The player movement can then be calculated based on the previously mentioned positional array.

I currently use 4 different functions to handle movement, though this could actually be improved significantly. The current code looks like this:

## Remove the previous player position
function player_clear () {
  cell="${m[${y}]:${x}:1}"
  tput cup ${y} ${x}
  [[ ${cell} == "${empty}" ]] && printf "${empty}"
  [[ ${cell} == "${prevdoor}" ]] && printf "${prevdoor}"
  [[ ${cell} == "${nextdoor}" ]] && printf "${nextdoor}"
  return 0
}
## Draw the current player position
function player_draw () {
  tput cup ${y} ${x}
  printf "${player}\b"
  tput cup $(( ${height} + 6 )) 0
  return 0
}
## Up Movement Checks
function player_movement_up () {
  declare direction
  direction="${m[$(( ${y} - 1 ))]:${x}:1}"
  key="Up"
  [[ ${direction} == "${wall}" ]] && return 1
  [[ ${direction} == "${nextdoor}" ]] && map_change_next
  [[ ${direction} == "${prevdoor}" ]] && map_change_prev
  player_clear
  tput cup $(( ${height} + 6 )) 0
  (( y-- ))
  player_position
  player_draw
  tput cup $(( ${height} + 6 )) 0
  return 0
}
## Down Movement Checks
function player_movement_down () {
  declare direction
  direction="${m[$(( ${y} + 1 ))]:${x}:1}"
  key="Down"
  [[ ${direction} == "${wall}" ]] && return 1
  [[ ${direction} == "${nextdoor}" ]] && map_change_next
  [[ ${direction} == "${prevdoor}" ]] && map_change_prev
  player_clear
  tput cup $(( ${height} + 6 )) 0
  (( y++ ))
  player_position
  player_draw
  tput cup $(( ${height} + 6 )) 0
  return 0
}
## Right Movement Checks
function player_movement_right () {
  declare direction
  direction="${m[${y}]:$(( ${x} + 1 )):1}"
  key="Right"
  [[ ${direction} == "${wall}" ]] && return 1
  [[ ${direction} == "${nextdoor}" ]] && map_change_next
  [[ ${direction} == "${prevdoor}" ]] && map_change_prev
  player_clear
  tput cup $(( ${height} + 6 )) 0
  (( x++ ))
  player_position
  player_draw
  tput cup $(( ${height} + 6 )) 0
  return 0
}
## Left Movement Checks
function player_movement_left () {
  declare direction
  direction="${m[${y}]:$(( ${x} - 1 )):1}"
  key="Left"
  [[ ${direction} == "${wall}" ]] && return 1
  [[ ${direction} == "${nextdoor}" ]] && map_change_next
  [[ ${direction} == "${prevdoor}" ]] && map_change_prev
  player_clear
  tput cup $(( ${height} + 6 )) 0
  (( x-- ))
  player_position
  player_draw
  tput cup $(( ${height} + 6 )) 0
  return 0
}
          

All movements register the global x and y variables (either +1 or -1, depending on the direction) and then redraws the player character. Before it does this though, it uses the backspace character and replaces it with the map arrays character to clear the players previous position.

The last thing would be level transition, which I initially wasn’t sure how to do. However, I now have a fairly decent system that, based on the previous door (-) and next door (+) coords, can put the player in a more seemless transition position.

Each map transition does require a full board redraw though.

## Change Map to the Next Level
function map_change_next () {
  [[ ! -f "${base}/maps/$(( ${map} + 1 )).map" ]] && title_draw 2 && main
  (( map++ ))
  unset m
  clear
  board_draw
  x=${cx[${map}]}
  y=${cy[${map}]}
  player_position
  player_draw
  player_controls
  return 0
}
## Change Map to the Previous Level
function map_change_prev () {
  [[ ! -f "${base}/maps/$(( ${map} - 1 )).map" ]] && return 0
  (( map-- ))
  unset m
  clear
  board_draw
  x=${px[${map}]}
  y=${py[${map}]}
  player_position
  player_draw
  player_controls
  return 0
}
          

These are both similar again, but the initial check is different. If, for some reason, map 1 has a previous door on it, the code wont do anything. It will act like the start doors in door. The next door, though, will display the “Game Over” screen if you try and go to map 4.

All in all, the actual functionality of the game is limited, but I’m pretty please with how the overall engine has come out (considering there aren’t libraries for this sort of thing).

I could use something like the ncurses library to do better drawing, but I feel that would detract too much from what I want to achieve.

You can find where I got to in the “brouge” repo on my git page.


PHP Like Strings in Bash

22 March 2020

PHP has a few nice features with regards to string functions that bash lacks. I decided to try and create functions that work almost exactly the same as the PHP counterpart. I didn’t actually look at how the PHP functions were created and instead just went off the relevant function documentation pages.

The first function I thought would be nice was substr; this is more flexible than the standard bash sub string notation. For example:

#!/bin/bash
START=6
END=8
STRING="this is a string!"
STRING=${STRING:${START}:${END}}
echo ${STRING}
          

This would output “s a stri”. Maybe what we are after, but maybe not. But that’s as far as we can go with direct sub strings. Anything else has to be calculated before hand. One of the primary benefits to how PHP handles sub strings is we can actually work backwards. For example:

#!/bin/bash
$STRING = substr ( "this is a string!", 6, -3 );
echo $STRING;
          

This will now give us the same result as out bash sub string; “s a stri”. Except, we didn’t know how long that sub string was going to be; we just knew that relative to the main string, 3 characters from the end is where we want to finish. of course we can continue to use 6 and 8 as the values in PHP and we’d get the same results, but by working backwards from the end of the main string, we open up a lot more possibilities.

Trying to simulate this is bash is trickier than you may imagine because of the base line we start with. To counter this, you have to do a fair amount of maths on the main string value. In the end I came up with this:

#!/bin/bash
string.substr () {
declare r t x y
[[ ! ${1} || ! ${2} ]] && return 1
[[ ! ${2} =~ ^[-0-9]+$ ]] && return 2
[[ ${3} && ! ${3} =~ ^[-0-9]+$ ]] && return 3
[[ ${2:0:1} == "-" ]] && r="0" || r="1"
[[ ${3:0:1} == "-" ]] && t="0" || t="1"
case ${r} in
  0)
    if [[ ! ${3} ]]; then
      printf "%s" "${1:$(( ${#1} - ${2:1} ))}"
    else
      case ${t} in
        0)
          x=$(( ${#1} - ${2:1} ))
          y=$(( ( ${#1} - ${3:1} ) - ( ${#1} - ${2:1} ) ))
          (( y <= 0 )) && y=${#1}
          printf "%s" "${1:${x}:${y}}"
        ;;
        1)
          x=$(( ${#1} - ${2:1} ))
          y=$(( ( ${#1} - ${2:1} ) - ( ${#1} - ${3} ) ))
          (( y >= 0 )) && y=${#1}
        printf "%s" "${1:${x}:${y}}" ;;
      esac
    fi
  ;;
  1)
    if [[ ! ${3} ]]; then
      printf "%s" "${1:${2}}"
    else
      case ${t} in
        0)
          y=$(( ${2} - ( ${#1} - ${3:1} ) ))
          (( y >= 0 )) && y="-${#1}"
          printf "%s" "${1:${2}:${y:1}}"
        ;;
        1) printf "%s" "${1:${2}:${3}}" ;;
      esac
    fi
  ;;
esac
return 0
}
          

This took a fair amount of thinking about the get it to behave exactly as PHP’s function does. However, it does all the heavy lifting for you, and allows you to write something that’s almost as elegant as PHP’s solution:

#!/bin/bash
STRING=$( string.substr "this is a string!" 6 8 )
echo ${STRING}
STRING=$( string.substr "this is a string!" 6 -3 )
echo ${STRING}
          

After feeling good about my success with sub strings, I decided to continue that trend with my next two functions. Again, I checked PHP’s implimentation of these and tried to follow suit:

#!/bin/bash
string.strpos () {
  declare -i i OFFSET
  [[ ! ${1} || ! ${2} ]] && return 2
  [[ ${3} && ! ${3} =~ ^[0-9]+$ ]] && return 3
  [[ ! ${3} && ${1} != *${2}* ]] && printf "%i" "-1" && return 1
  [[ ${3} && ${1:${3}} != *${2}* ]] && printf "%i" "-1" && return 1
  [[ ${3} ]] && OFFSET="${3}" || OFFSET="0"
  for (( i=$(( 0 + OFFSET )); i<${#1}; i++ )); do
    [[ ${1:${i}:${#2}} == ${2} ]] && printf "%i" "$(( ${i} + 1 ))" && break
  done
  return 0
}
string.strrchr () {
  declare -i i POS
  [[ ! ${1} || ! ${2} ]] && return 2
  [[ ${1} != *${2}* ]] && printf "%i" "-1" && return 1
  for (( i=0; i<${#1}; i++ )); do
    [[ ${1:${i}:${#2}} == ${2} ]] && POS=$(( ${i} + 1 ))
  done
  printf "%i" "${POS}"
  return 0
}
          

Just like PHP, these functions will search a string for the first and last instances of a second substring. Trying to work out how to iterate over the characters in a string took most of the time, but once I’d figured it out it was pretty easy. I even included the offset ability of PHP. These allow you to do something like this:

#!/bin/bash
STR[0]=$( string.strpos "string some text string" "str" )
STR[1]=$( string.strrchr "string some text string" "str" )
STR[2]=$( string.strpos "string some text string" "str" 1 )
echo ${STR[0]}
echo ${STR[1]}
echo ${STR[2]}
          

Which give you the results 1, 18, 18.

So, with combining the functions above, you can create substrings based on search results of a main string. As usual you can find these added to bashlib on my git page. I’ve also added a couple of other prototypes (like file.sock) but that will be for another day (also, enjoy the load of conversion functions I’ve added — not sure how useful they really are though).


Batch File Filtering

17 February 2020

I had some client work I needed to sort out, which is why this post is about Windows batch files.

The need was to remove lines of a text file (specifically a csv file) based on the contents of another. Both files where laid out the same (in the format of Name,Email,ID) and the secondary file essentially worked as a duplicate list, I just needed to delete them from the primary list.

In bash, this would be pretty trivial, but batch files are significantly less …advanced? Regardless, it took a while to work out exactly how to read files line by line and what I came up with is no doubt not optimal, but honestly, it does what it needs to do and it was 1AM by the time I’d already sorted it out. It essentially works backwards. It copies the master file, then line by line of the filter file removes any lines that match from the output

I added a bit of extra functionality after finishing, so you can input files via the command line arguments, and it does a proper check for files existing.

@echo off
rem delay variable expansion
setlocal enabledelayedexpansion
rem read command line arguments
if "%1" == "" (
  set LIST_TO_FILTER=master.csv
) else (
  set LIST_TO_FILTER=%1
)
if "%2" == "" (
  set LIST_TO_FILTER_FROM=filter.csv
) else (
  set LIST_TO_FILTER_FROM=%2
)
if "%3" == "" (
  set OUTPUT=output.csv
) else (
  set OUTPUT=%3
)
rem count variable
set /A COUNT=0
rem print files and wait for confirmation
echo.
echo File to Filter: %LIST_TO_FILTER%
echo Filter From: %LIST_TO_FILTER_FROM%
echo Final Output: %OUTPUT%
echo.
echo ---------------
echo.
echo Press any key to start...
@pause >nul 2>&1
rem check if files exists
if not exist %LIST_TO_FILTER% (
  echo %LIST_TO_FILTER% does not exist
  echo Press any key to abort...
  @pause >nul 2>&1
  exit /B
)
if not exist %LIST_TO_FILTER_FROM% (
  echo %LIST_TO_FILTER_FROM% does not exist
  echo Press any key to abort...
  @pause >nul 2>&1
  exit /B
)
rem copy LIST_TO_FILTER to OUTPUT
@copy %LIST_TO_FILTER% %OUTPUT% >nul 2>&1
rem read LIST_TO_FILTER_FROM line by line
echo|set /p="Filtering"
for /f "tokens=*" %%A in (%LIST_TO_FILTER_FROM%) do (
  rem find line in OUTPUT and copy all but that line to OUTPUT.tmp
  find /v "%%~A" < %OUTPUT% > %OUTPUT%.tmp
  rem move OUTPUT.tmp back to OUTPUT
  @move %OUTPUT%.tmp %OUTPUT% >nul 2>&1
  rem count lines
  set /A COUNT+=1
  rem print indicator
  echo|set /p="."
)
rem print final processing count and end
echo.
echo Done processing %COUNT% lines
echo.
echo Press any key to close...
@pause >nul 2>&1
          

Bash Classes

30 January 2020

I’m working on a pesudo class module for bashlib.

Currently it’s going really well, I wont ever be able to get the full range of class abilities in Bash, what I have can be used in a rudementary way.

#!/bin/bash
source ~/bin/bashlib/bashlib.sh
bashlib.load "class"
User__construct () {
  this : name="No_Name"
  this : dob="1970-01-01"
  this : phone="11111111111"
  this : notes="N/A"
}
User_GetInfo () {
  echo "User  --" $( this . )
  echo "Name:     " $( this - name )
  echo "DOB:      " $( this - dob )
  echo "Phone:    " $( this - phone )
  echo "Notes:    " $( this - notes )
}
USERS=( "bob"           "jim"           "sarah"         "jane"          "joe"           "noname" )
NAMES=( "Wight, Bob"    "Bishop, Jim"   "Amis, Sarah"   "Smith, Jane"   "Blogs, Joe" )
DATES=( "1972-11-18"    "1975-08-23"    "1979-02-05"    "1968-04-22" )
PHONE=( "01189434321"   "01189693219"   "01189679234" )
for (( i=0; i < ${#USERS[@]}; i++ )); do
  class.new User "${USERS[${i}]}" { name="${NAMES[${i}]}" dob="${DATES[${i}]}" phone="${PHONE[${i}]}" }
done
          

This takes the information from NAMES/DATES/PHONE arrays and puts them in to classes based on the USERS array. So, you could easily do:

bob : GetInfo
jim : GetInfo
sarah : GetInfo
jane : GetInfo
joe : GetInfo
noname : GetInfo
          

Which would then print out the details provided by the previous arrays. The good thing about this class function, is that the CLASS__construct function runs initially. Which means that you can set default variables, instead of relying on the class.new function filling them all in. This is apparent with the users “jane” “joe” and “noname”, which have an increasing amount of data ommited from their arrays. Rather than just being blank though, they will take the defaults set in User__construct. With a simple tweak to the for loop above, we can print the information out:

for (( i=0; i < ${#USERS[@]}; i++ )); do
  class.new User "${USERS[${i}]}" { name="${NAMES[${i}]}" dob="${DATES[${i}]}" phone="${PHONE[${i}]}" }
done
for (( i=0; i < ${#USERS[@]}; i++ )); do
  class.new User "${USERS[${i}]}" { name="${NAMES[${i}]}" dob="${DATES[${i}]}" phone="${PHONE[${i}]}" }
  ${USERS[${i}]} : GetInfo
  (( i != ${#USERS[@]} - 1 )) && echo "=========="
done
          

This will give the results:

% bash class.test.sh
User  -- bob
Name:    Wight, Bob
DOB:     1972-11-18
Phone:   01189434321
Notes:   N/A
==========
User  -- jim
Name:    Bishop, Jim
DOB:     1975-08-23
Phone:   01189693219
Notes:   N/A
==========
User  -- sarah
Name:    Amis, Sarah
DOB:     1979-02-05
Phone:   01189679234
Notes:   N/A
==========
User  -- jane
Name:    Smith, Jane
DOB:     1968-04-22
Phone:   11111111111
Notes:   N/A
==========
User  -- joe
Name:    Blogs, Joe
DOB:     1970-01-01
Phone:   11111111111
Notes:   N/A
==========
User  -- noname
Name:    No_Name
DOB:     1970-01-01
Phone:   11111111111
Notes:   N/A
          

As I progress with the implimentation, I’ll do my best to explain in more detail. But hopefully that gives you some idea of what to expect.

One of the key features of this new class system is the contextual “this” function. It’s incredibly difficult to actually return the true name of a function in bash. The FUNCNAME array might contain it, if it’s called in the right way. But a more surefire way is to simply pass the custom FUNC_NAME variable around (without the user needing to). This allows for a pretty powerful set of commands that you can run by calling “this” instead of needing to use specific names.

In the above example we use 3 of the 4 currently avaliable commands:

this .                  # . Displays the name of the current class
          this : variable=value   # : Sets variables
          this - variable         # : Prints a variable value
          this @ FunctionName     # @ runs a self named function (eg MyClass_FunctionName)
          

This allows you to set variables, run other functions all from within a class function and without needing outside help to determine which class it should affect.

The above would be similar to itterating over this (still in a class setting):

sName="joe"
echo ${sName}                           # . Equivilant
${sName} : variable=value               # : Equivilant
${sName} : variable                     # - Equivilant
this @ FunctionName                     # @ runs a self named function (eg MyClass_FunctionName)
            

While for a single set, that works fine; if you need two users, Joe and Jane, you would need to type a second set of calls. And then for three, four, five etc.