A place for miscellaneous odds & ends.
Useful Config Files
Updated Jan 2023
Config files for Bash, (Neo)Vim, Gnome keyboard bindings, and more are now stored on my Github account:
github.com/JacobCrabill/config
Config files for Bash, (Neo)Vim, Gnome keyboard bindings, and more are now stored on my Github account:
github.com/JacobCrabill/config
Working on Remote Clusters
Python Virtual Environments
When working on a cluster or other desktop environment you don't have admin access to, there's a good chance that the version of Python (or its packages, or the libraries those packages are compiled against...) aren't exactly what you need. Virtual environments are a simple way to get your own local version of Python with all the packages you need.
Python 3.5+ comes ready with a venv script. Just do the following after compiling your own (local) version of Python:
python3 -m venv /path/to/new/venv/
After that, you activate the virtual environment with:
. /path/to/new/venv/bin/activate
which is a handy thing to stick an 'alias' into your .bashrc for.
Once the venv is activated, you can install all the Python goodness you want with "pip install __stuff__".
When working on a cluster or other desktop environment you don't have admin access to, there's a good chance that the version of Python (or its packages, or the libraries those packages are compiled against...) aren't exactly what you need. Virtual environments are a simple way to get your own local version of Python with all the packages you need.
Python 3.5+ comes ready with a venv script. Just do the following after compiling your own (local) version of Python:
python3 -m venv /path/to/new/venv/
After that, you activate the virtual environment with:
. /path/to/new/venv/bin/activate
which is a handy thing to stick an 'alias' into your .bashrc for.
Once the venv is activated, you can install all the Python goodness you want with "pip install __stuff__".
NUMA Control: Bind MPI Ranks to Specific Hardware
This is really important if you want to run MPI with OpenMP, or just make sure you're using the nearest CPU to a GPU or other accelerator (for instance, each ICME cluster node has 2 6-core processors, each of which is bound to two GPUs; naive 'mpirun -n 4' won't bind to the nearest CPU for each GPU).
Example: Profile an MPI application using nvprof, and bind each rank to a separate physical CPU:
This is really important if you want to run MPI with OpenMP, or just make sure you're using the nearest CPU to a GPU or other accelerator (for instance, each ICME cluster node has 2 6-core processors, each of which is bound to two GPUs; naive 'mpirun -n 4' won't bind to the nearest CPU for each GPU).
Example: Profile an MPI application using nvprof, and bind each rank to a separate physical CPU:
mpirun --bind-to none -n 2 omp_run.sh
omp_run.sh:
export OMP_NUM_THREADS=6
rank=$OMPI_COMM_WORLD_LOCAL_RANK
if [ $rank == 0 ]; then
numactl --cpunodebind=0 nvprof -o nv.%p.%h.out my_executable
else
numactl --cpunodebind=1 nvprof -o nv.%p.%h.out my_executable
fi
When using numactl, the "node" being bound to is the CPU socket; what node of a cluster you're running on is a separate issue. But, the OMPI_COMM_WORLD_LOCAL_RANK will give you the node-local MPI rank for your job, so this is about all you need to control processing within a single node.
TODO: NVIDIA GPU visible device masking
TODO: NVIDIA GPU visible device masking
HPC Profiling
IPM, perf counters, nvprof + nvvp
<TODO>
NVIDIA Visual Profiler:
Profile CUDA applications on NVIDIA GPUs
mpirun -n # nvprof -o nv.%p.nvprof my_app
Then import the *.nvprof into NVVP, making sure to select the "multiple processes" option along the way
IPM, perf counters, nvprof + nvvp
<TODO>
NVIDIA Visual Profiler:
Profile CUDA applications on NVIDIA GPUs
mpirun -n # nvprof -o nv.%p.nvprof my_app
Then import the *.nvprof into NVVP, making sure to select the "multiple processes" option along the way
Handy List of 'Roll-Your-Own' Software for Remote Environments
<TODO>
HDF5, serial or parallel versions..
SWIG
METIS
OpenBLAS (make sure to compile using your preferred compiler or mpicxx/mpicc will wrap the wrong compiler!)
<TODO>
HDF5, serial or parallel versions..
SWIG
METIS
OpenBLAS (make sure to compile using your preferred compiler or mpicxx/mpicc will wrap the wrong compiler!)
Annoying MPI / Python Valgrind Error Suppression
Wrapping codes with Python lets you do some pretty cool stuff. However, once you go down the Python route, running Valgrind on your wrapped code produces pretty much garbage due to Python's redefinition of malloc and free. Thankfully, if you don't want to reompile Python from source with memcheck debugging enabled, you can use a handy suppression file to do the trick (just uncomment all the PyObject_Free / PyObject_ReAlloc lines):
https://svn.python.org/projects/python/trunk/Misc/valgrind-python.supp
To use this (or other) suppression files with valgrind, just use the suppresion option, which can be used multiple times:
Wrapping codes with Python lets you do some pretty cool stuff. However, once you go down the Python route, running Valgrind on your wrapped code produces pretty much garbage due to Python's redefinition of malloc and free. Thankfully, if you don't want to reompile Python from source with memcheck debugging enabled, you can use a handy suppression file to do the trick (just uncomment all the PyObject_Free / PyObject_ReAlloc lines):
https://svn.python.org/projects/python/trunk/Misc/valgrind-python.supp
To use this (or other) suppression files with valgrind, just use the suppresion option, which can be used multiple times:
valgrind --suppressions=valgrind-python.supp --suppressions=other-supp-file.supp my.exe
OpenMPI gives a similarly long list of useless memcheck errors upon startup; use this file here to suppress most, if not all, of the errors:
{
Conditional jump depends on uninitalized values
Memcheck:Cond
...
fun:mca_bml_base_init
}
{
Invalid read of size 8
Memcheck:Value8
...
fun:mca_bml_base_init
}
{
Syscall param write(buf) points to uninitialised byte(s)
Memcheck:Param
write(buf)
...
fun:mca_bml_base_init
}
{
Conditional jump depends on uninitalized values
Memcheck:Cond
fun:memcpy
...
fun:mca_bml_r2_component_init
}
{
Syscall param write(buf) points to uninitialised byte(s)
Memcheck:Param
write(buf)
...
fun:mca_bml_r2_component_init
}
{
Invalid read of size 8
Memcheck:Value8
...
fun:ompi_mpi_init
}
{
Syscall param write(buf) points to uninitialised byte(s)
Memcheck:Param
write(buf)
...
fun:ompi_mpi_init
}
{
Conditional jump depends on uninitalized values
Memcheck:Cond
...
fun:ompi_mpi_init
}
BASH / Unix Trickery
BASH Trickery
1) Applying a bash command to all files matching a certain pattern in a directory:
1) Applying a bash command to all files matching a certain pattern in a directory:
1 #!/bin/bash
2 # Find all PyFR solution files matching *G0-[number].pyfrs and place iteration numbers into an array
3 ITER_LIST=($(ls | grep -oP '(?<=G0-)(([0-9]*)(?=\.pyfrs))' | sort -n))
4
5 # Number of values in array
6 N_FILES=${#ITER_LIST[@]}
7
8 for iter in "${ITER_LIST[@]}"
9 do
10 # Run command 'my_exe' with arugments 'args' along with each number in the list
12 COMMAND="my_exe args $iter"
13 echo $COMMAND
14 $COMMAND
15 done
2) Keep your SLURM allocation running for extra commands (in case your job fails unexpectedly):
1 #!/usr/bin/bash
2 #SBATCH
3
4 srun -n $SLURM_NTASKS my_exe
5
6 KEEP_RUNNING=1
7 while [ $KEEP_RUNNING -eq 1 ]; do
8 echo "Run completed - awaiting further input into other_commands.sh."
9 . other_commands.sh
10 sleep 2m;
11 done
Useful BASH / grep / sed commands
Replace numbers in scientific notation (positive OR negative exponent) on lines of all *.vtu files starting with "<!-- TIME ":
Replace numbers in scientific notation (positive OR negative exponent) on lines of all *.vtu files starting with "<!-- TIME ":
sed -i '\_<\!--\ TIME\ _s_[0-9]\.[0-9]*e[+-][0-9]*_0.0e+0_g' *.vtu
(Note the use of underscores instead of slashes as the sed delimiter for clarity)
Get a list of iteration #'s from a directory full of plot files, remove leading zeros, and remove any which have already been saved to png's:
Get a list of iteration #'s from a directory full of plot files, remove leading zeros, and remove any which have already been saved to png's:
ITER_LIST=($(ls $CASE_DIR | grep -oP '(?<=fk0_000)(([0-9]*)(?=\.pvtu))' | tr ' ' '\n' | sort -n))
PNG_LIST=($(ls PNG/ | grep -oP '(?<=LIC.)(([0-9]*)(?=\.png))' | sort -n))
ITER_LIST=($(echo ${ITER_LIST[@]} ${PNG_LIST[@]} | tr ' ' '\n' | sort | uniq -u))
Useful awk / find / etc. commands
Find all static libraries (*.a), show their size (ls -alh), pick out the size & name columns, and sort by size
find . -name "*.a" | xargs ls -alh | awk '{print $5 " " $9}' | sort -nk1
Misc. Ubuntu Tips
Getting a USB Webcam Working
Bought a cheap USB webcam off of Amazon ($30), and although the video worked out-of-the-box, the microphone seemingly did not - the volume was so quiet I thought the microphone was just not enabled at all.
The fix turned out to be quite simple in the end (see: forum.level1techs.com/t/very-very-quiet-mic-with-any-distro/145386/8)
Bought a cheap USB webcam off of Amazon ($30), and although the video worked out-of-the-box, the microphone seemingly did not - the volume was so quiet I thought the microphone was just not enabled at all.
The fix turned out to be quite simple in the end (see: forum.level1techs.com/t/very-very-quiet-mic-with-any-distro/145386/8)
- Install and open alsamixer
- Hit "F6" to show all connected sound devices, and select the USB webcam gadget
- This may show a message about no suitable devices - this is what tripped me up for a long time. Note at the top, the "View:" line probably has "Playback" selected. Hit "F4" to switch to "Capture" (i.e. microphone)
- Hit the up arrow key to boost the gain on the microphone which has hopefully appeared
Useful Software in Ubuntu
Programming (C/C++)
Document Creation
Texmaker - very good environment for building Latex documents; but first, get the full Latex distribution from Texlive: https://www.tug.org/texlive/
Ubuntu desktop mods (14.04, possibly 16.04)
My (current) favorite flavor of Linux: Elementary OS. Basically a mod of Ubuntu 14.04 with a simple, clean desktop environment without all the awful over-the-top eye-candy resource-hog features that was Unity.
Video Playback
Just discovered "melt" - simple terminal-based VLC-like player that can do frame-by-frame stepping, forwards and backwards (besides apparently being a fully-featured video editing engine). Appears to come default with Ubuntu.
PDF Creation from images
If the images came from pictures you took, use ScanTailor to crop, unskew & dewarp the images, then use convert:
convert *.tif out.pdf
- QT Creator: I uses QT Creator for all of my software development, because although it's meant for QT development, it's equally applicable to C/C++ projects, and it has excellent "intellisense"-style features that rival those of MS Visual Studio. Unlike Visual Studio, however, it is not a complete resource hog that takes forever to load, and it's also free and platform-independent.
- Code::Blocks: Definitely among the best; has lots of good features & is designed for C++ development.
Document Creation
Texmaker - very good environment for building Latex documents; but first, get the full Latex distribution from Texlive: https://www.tug.org/texlive/
Ubuntu desktop mods (14.04, possibly 16.04)
- Ubuntu Tweak - Allows nice features like "hot corners", which shows your workspaces, windows, or desktop with the swipe of your mouse into a corner of the screen (a feature which I can no longer live without!)
- Compiz Fusion Icon - Allows modifications to the Gnome desktop, including the window manager. Purely for eye-candy purposes, but nice nonetheless.
- MenuLibre (menulibre in package manager) - Replaces (?) the old "alacarte" desktop/application-launcher editor
My (current) favorite flavor of Linux: Elementary OS. Basically a mod of Ubuntu 14.04 with a simple, clean desktop environment without all the awful over-the-top eye-candy resource-hog features that was Unity.
Video Playback
Just discovered "melt" - simple terminal-based VLC-like player that can do frame-by-frame stepping, forwards and backwards (besides apparently being a fully-featured video editing engine). Appears to come default with Ubuntu.
PDF Creation from images
If the images came from pictures you took, use ScanTailor to crop, unskew & dewarp the images, then use convert:
convert *.tif out.pdf
Ubuntu Tweaks
Workspace Grid
Install the Gnome tweaks tool (sudo apt-get install gnome-tweak-tool) and the Chrome Gnome plugin (sudo apt-get install chrome-gnome-shell)
Install the Gnome workspace-grid extension from their website
Setup keyboard shortcuts for the workspace-grid extension (once enabled via Gnome Tweaks):
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-up "['<Super>Up']"
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-down "['<Super>Down']"
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-left "['<Super>Left']"
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-right "['<Super>Right']"
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-up "['<Ctrl><Shift><Super>Up']"
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-down "['<Ctrl><Shift><Super>Down']"
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-left "['<Ctrl><Shift><Super>Left']"
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-right "['<Ctrl><Shift><Super>Right']"
gsettings set org.gnome.shell.extensions.dash-to-dock scroll-action 'cycle-windows'
Install the Gnome tweaks tool (sudo apt-get install gnome-tweak-tool) and the Chrome Gnome plugin (sudo apt-get install chrome-gnome-shell)
Install the Gnome workspace-grid extension from their website
Setup keyboard shortcuts for the workspace-grid extension (once enabled via Gnome Tweaks):
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-up "['<Super>Up']"
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-down "['<Super>Down']"
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-left "['<Super>Left']"
gsettings set org.gnome.desktop.wm.keybindings switch-to-workspace-right "['<Super>Right']"
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-up "['<Ctrl><Shift><Super>Up']"
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-down "['<Ctrl><Shift><Super>Down']"
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-left "['<Ctrl><Shift><Super>Left']"
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-right "['<Ctrl><Shift><Super>Right']"
gsettings set org.gnome.shell.extensions.dash-to-dock scroll-action 'cycle-windows'
QtCreator Stuff
.config/QtProject/QtCreator.ini
^ Set FontSizeFollowsEditor=true to make the overall environment's font size actually readable on high-res screens
.config/QtProject/QtCreator.ini
^ Set FontSizeFollowsEditor=true to make the overall environment's font size actually readable on high-res screens
Post-Processing / Visualization
Create a movie from a set of PNGs that can be played on any device:
ffmpeg -framerate 15 -start_number 0000 -i Vis.%04d.png -pix_fmt yuv420p -r 15 -vf scale=1000:-2 Vis.mp4
Convert mp4 to animated gif:
ffmpeg -framerate 15 -start_number 0000 -i Vis.%04d.png -pix_fmt yuv420p -r 15 -vf scale=1000:-2 Vis.mp4
Convert mp4 to animated gif:
- Generate a palette:
ffmpeg -y -ss 30 -t 3 -i input.flv \ -vf fps=10,scale=320:-1:flags=lanczos,palettegen palette.png - Output the GIF using the palette:
ffmpeg -ss 30 -t 3 -i input.flv -i palette.png -filter_complex \ "fps=10,scale=320:-1:flags=lanczos[x];[x][1:v]paletteuse" output.gif
Useful tips for ElementaryOS
Duplicate icons in Plank
Assuming you have the .desktop file in /usr/share/applications, do the following:
Assuming you have the .desktop file in /usr/share/applications, do the following:
- Open the affected application
- Open a new terminal, type "xprop", and then click on the application window
- On the output that appears, find the line that starts with "WM_CLASS(STRING)" and down the first string (without quotations) that follows
- Copy the .desktop file from /usr/share/applications to ~/.local/share/applications
- If not already present, add a line starting with "StartupWMClass="
- Type in the WM_CLASS string you previously found; e.g. "StartupWMClass=Google-chrome-stable"
- If the icon is currently present in Plank, close the application, right-click the icon, and deselect "keep in dock"
- Re-open the application, right-click the icon, and re-enable "keep in dock"
- Close & re-open the application from the dock to be sure you get just one icon
Installing Matlab Student Edition
The student version of Matlab only comes in 32bit, and it cannot make use of the 64bit Java Runtime Environment (JRE) on 64bit machines. To fix this, you need to install 32bit versions of the JRE libraries and such for Matlab to make use of. See here for details, but basically you should be able to just do the following from the terminal:
sudo apt-get install libxtst6:i386 libXext6:i386 libxi6:i386 libncurses5:i386 libxt6:i386 libxpm4:i386 libxmu6:i386 libxp6:i386
After installing these, Matlab *should* be able to be installed by something like the following:
sudo /media/MATHWORKS_R2012A/install -glnx86 -v
Once installed, you'll probably have to run Matlab as sudo and give it the flag -glnx86.
The student version of Matlab only comes in 32bit, and it cannot make use of the 64bit Java Runtime Environment (JRE) on 64bit machines. To fix this, you need to install 32bit versions of the JRE libraries and such for Matlab to make use of. See here for details, but basically you should be able to just do the following from the terminal:
sudo apt-get install libxtst6:i386 libXext6:i386 libxi6:i386 libncurses5:i386 libxt6:i386 libxpm4:i386 libxmu6:i386 libxp6:i386
After installing these, Matlab *should* be able to be installed by something like the following:
sudo /media/MATHWORKS_R2012A/install -glnx86 -v
Once installed, you'll probably have to run Matlab as sudo and give it the flag -glnx86.
GiMMiK .c / .cu file generator
Uses GiMMiK to create a C or CUDA file from *.txt files containing a unique ID followed by an operator matrix [Change the extension to .py and use with Python 3].
|
|
PyFR Mesh Transformation Script
Because I'll forget where I put it eventually, here's a script that reads in a (3D) PyFR (.pyfrm) mesh file, and gives you the option of either replacing the UUID, translating the grid, or applying a scaling factor to it (the initial file is overwritten) (read/write using the C++ HDF5 API; download & install that on your own).
g++ -std=c++11 scalepyfr.cpp -o scalePyFR -lhdf5 -lhdf5_cpp
g++ -std=c++11 scalepyfr.cpp -o scalePyFR -lhdf5 -lhdf5_cpp
scalepyfr.cpp | |
File Size: | 4 kb |
File Type: | cpp |
PyFR to Gmsh Grid Converter
Again, because I'll forget where I put it eventually, here's a script that reads in a (2D or 3D) PyFR (.pyfrm) mesh file (quads or hexas only), and converts the file to an ASCII Gmsh .msh file. Up to 5th-order quads/hexas are available, though the file can be easily modified to output higher-order elements. Again, you'll need a version of the C++ HDF5 library.
g++ -std=c++11 pyfr2gmsh.cpp -o pyfr2gmsh -lhdf5 -lhdf5_cpp
g++ -std=c++11 pyfr2gmsh.cpp -o pyfr2gmsh -lhdf5 -lhdf5_cpp
|
|
2D NACA Airfoil Mesh in Gmsh
I spent a while searching the internet for a good NACA0012 mesh in Gmsh format in the first place (and then further time refining it into a mesh that would be potentially useful with HiFiLES), so I feel I should return the favor in the hopes that somebody finds this as useful as I did. Below are 2 .geo files (input files for Gmsh) which create a) a largely unstructured mesh of triangles, and b) a completely structured mesh of quads. In both cases, it should be fairly easy to modify the number of points/elements in the mesh as desired.
|
|
SU2 to Gmsh Grid Converter
I recently (well, back in 2015 I think) came across the need to convert a grid in the SU2 format to Gmsh format for use in one of my codes. Since this seems like something that might come in handy in the future, I took a few hours and wrote this little script here to solve the problem.
Either use the pre-compiled binary, or compile from source as so:
g++ -std=c++11 sU2_to_gmsh.cpp -o su2gmsh
Either use the pre-compiled binary, or compile from source as so:
g++ -std=c++11 sU2_to_gmsh.cpp -o su2gmsh
|
|
Miscellaneous
Install CruiseControl.rb on Ubuntu 12.04
This caused a couple hours of pain and frustration. Installing CruiseControl.rb requires Ruby, which doesn't seem to enjoy being installed on Ubuntu, for whatever reason (specifically the Ruby debugger, it seems). Instead, install a whole CruiseControl.rb deb package via apt-get here:
http://blog.tryphon.org/alban/2009/02/21/install-cruisecontrolrb-on-debian-or-ubuntu/
The above link explains how to setup & use CruiseControl. The necessary apt repositories to install the software are located here:
http://debian.tryphon.eu/
Don't miss the step to add the repository's key to you apt client! (On the above page, just below the Debian release listings). Works on 12.04; does not appear to work on 13.04.
This caused a couple hours of pain and frustration. Installing CruiseControl.rb requires Ruby, which doesn't seem to enjoy being installed on Ubuntu, for whatever reason (specifically the Ruby debugger, it seems). Instead, install a whole CruiseControl.rb deb package via apt-get here:
http://blog.tryphon.org/alban/2009/02/21/install-cruisecontrolrb-on-debian-or-ubuntu/
The above link explains how to setup & use CruiseControl. The necessary apt repositories to install the software are located here:
http://debian.tryphon.eu/
Don't miss the step to add the repository's key to you apt client! (On the above page, just below the Debian release listings). Works on 12.04; does not appear to work on 13.04.